Free SE video lectures from MIT

April 16, 2009 | Leave a Comment

Course Description

This course covers topics on the engineering of computer software and hardware systems: techniques for controlling complexity; strong modularity using client-server design, virtual memory, and threads; networks; atomicity and coordination of parallel activities; recovery and reliability; privacy, security, and encryption; and impact of computer systems on society. It also looks at case studies of working systems and readings from the current literature provide comparisons and contrasts, and do two design projects.

http://academicearth.org/courses/computer-system-engineering

Posted by | Filed Under Lectures, Networking 

Network port configurations for MSDTC

July 27, 2008 | Leave a Comment

What is MSDTC?

MSDTC is the Microsoft Distributed Transaction Coordinator, which is a transaction manager program that permits client applications to include several different sources of data into one transaction. MSDTC then coordinates committing the transaction across all of the servers that are listed in the transaction. MSDTC runs on all Windows operating systems, and is also installed by a variety of Microsoft applications, including Personal Web Server and SQL Server.

What Network Ports are Used?

MSDTC uses a number of TCP network ports for sending and receiving messages. This fact must be considered when MSDTC is running in a network environment where the servers involved in the transactions. Say you are running a multi-tier application, and each tier is separated by a router or firewall for security purposes. An example would be an application server in one tier communicating with a SQL Server database You will need to know what port numbers need to be opened for MSDTC transaction information to be able to pass through successfully.

For sending out transaction messages, MSDTC always uses the same TCP port - 135. Dealing with the response message is a little more tricky. MSDTC response messages return on a dynamically assigned port anywhere in a range from 1024 - 5000.

Configuring the MSDTC Respone Port Range

As most of you can probably guess, network administrators are not very fond of opening a wide range of ports all at once. So in order for MSDTC communications to still work and keep the network administrator happy at the same time, you will need to reduce the port range used by the response messages. This change is configured in the registry of the servers involved in the MSDTC communications. You will need to add a couple of keys to the registry.

Note: Please make sure to always take a backup of the registry prior to making any changes!

The location of the change is:

HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\Internet

The following entries need to be added:

Ports : REG_MULTI-SZ : 1024-1054

PortsInternetAvailable : REG_SZ : Y

UseInternetPorts : REG_SZ : Y

(There is a space both before and after each colon)

In this particular example we are limiting the responses to a port range of only 1024-1054. The exact range of ports to use is basically up to each individual organization.

Testing MSDTC Communications

So how do you know your changes were successful? Microsoft provides a handy little tool called DTCPing.exe that you can use to test MSDTC communications between servers. DTCPing can be downloaded from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=5e325025-4dcd-4658-a549-1d549ac17644&displaylang=en

This file is a self-extracting zip file.  Confusingly, the zip file and actual executable file are both named “DTCPing.exe”, so you need to make sure to extract to a separate directory, otherwise you will receive a ‘Cannot create output file’ error message.

Once extracted, simply launch DTCPing.exe. The tool must be up and running on both the sending and receving servers, otherwise the test will fail. The initial screen of the tool will look like this:

DTC Ping





From here just type in either the NetBIOS name or IP Address of the remote node and click Ping. Test messages will appear in the DTCPing windows of both the sending and responding servers, and a summary of the test will be presented at the end. The test scenario will also be written to a log file that can be found in the same directory as the DTCPing.exe file.

Posted by | Filed Under Windows, Networking, Troubleshooting 

Building a MSMQ Cluster

June 14, 2008 | Leave a Comment

 

Why?

Why bother clustering MSMQ at all?  Fault Tolerance.  As you probably know, Microsoft Message Queuing (MSMQ) is a queuing system which is typically used as middle-ware between applications.  MSMQ can store almost any type of data within MSMQ messages, and route them to another MSMQ queue manager where they can be retrieved by the receiving application.  MSMQ already provides an extra layer of reliability in the case where something on the back-end may not be available, the front-end can continue to receive requests and store them as MSMQ messages.  So what happens if the computer running MSMQ goes down, and the entire machine becomes unavailable?  That’s where a cluster helps us, as it adds provides fault-tolerance for those types of scenarios.  Node #1 fails, and Node #2 automatically takes it place.  An extremely simple and relatively inexpensive (given today’s hardware costs) solution. 

 

How to Cluster MSMQ?

All you need to cluster MSMQ in addition to the hardware and basic cluster setup, is a network name resource, and a disk-resource.  Step-by-step instructions can be found in the white-paper titled Deploying MSMQ 3.0 in a Server Cluster.

 

How Many Instances?  How does my Application Distinguish Between Local and Clustered MSMQ?

After you’ve successfully clustered MSMQ, it’s important to understand the difference between local and clustered MSMQ.  By default the local instance of the Message Queuing Service is set to Manual after MSMQ has been clustered, but it’s still there, and can easily be started at any time.  If you try running an MSMQ Application from the local command prompt, it’s either going to use the local MSMQ Service, or if that service isn’t started, it’s probably going to generate an error.

For most people in a typical two-node cluster, there are three instances of MSMQ.  Two local instances (one on each node) and the clustered instance which runs on the Active Node.  The clustered instance an be failed over between nodes from Active to Passive and vice-versa.  There aren’t always three instances though, a better way of stating that you can have (# of cluster nodes) + (# of clustered instances of MSMQ) nodes.

So, if there are so many available instances of MSMQ, how do you get your application to use a particular instance?  The answer depends on the the computer name that the application sees as its native computer name.  If you launch a regular command prompt and type in “ set COMPUTERNAME” it should display the current computer name.  If this is the name of the local node, then the application will try to use the local instance of MSMQ.  In order to get something to run within the context of the cluster, it needs to be using the same Network Name resource as clustered MSMQ.  This can be accomplished in one of two ways:

-Cluster your application in cluster administrator, ensuring it has a dependency on the same Network Name Resource being used by clustered MSMQ.  Then select the “Use Network Name as Computer Name” option

-Create a clustered command prompt.  Basically create a clustered resource for cmd.exe, and ensure that you follow the rules above and create a dependency on the same Network Name Resource being used by MSMQ, and ensure that the “Use Network Name as Computer Name” option is selected.  When you bring the resource online, you should see the command prompt.  From this clustered command prompt, try the same “set COMPUTERNAME” test I mentioned above.  This time you should see the virtual cluster name being displayed instead of the local node name.  If you run your application from this clustered command prompt, it should see clustered MSMQ as it’s native instance.  You can also run Computer Management (compmgmt.msc) from this prompt to administer clustered MSMQ.  Be aware that the prompt will not be displayed if you aren’t either physically at the node, or if you’re not logged in using the /console switch through mstsc.

 

Administering Clustered MSMQ

In order to properly administer clustered MSMQ, it’s recommended to either run Computer Management from a clustered command prompt (see above), or otherwise you can use the MMCV utility.  MMVC will allow you to launch Computer Management using a particular computer name, so you can easily specify the name of your virtual clustered server.  The syntax to use is “mmcv.exe -s <myVirtualServerName>”, it’s an extremely useful tool, and I’d strongly suggest including a copy of it on all of your MSMQ cluster nodes.  You can find out more about MMCV and find a link with a download in the following article: How to use the Mmcv.exe utility to manage clustered Message Queuing resources

Helpful Tips

Most clusters have multiple IPs, use the BindInterfaceIP registry key with clustered MSMQ to ensure that MSMQ binds to the correct IP: A cluster node with two network cards does not receive messages

Ensure that the local instances of MSMQ are set to run using the Local System account, even if they’re stopped and set to manual.  Having them set to use a particular user in an environment running clustered MSMQ will most likely cause issues.

If you’re going to be using Public Queues, or will have clustered MSMQ installed with the Active Directory Integration option, then ensure that Kerberos is enabled on your Network Name Resource in Cluster Administrator.  This will create an object for the virtual machine in Active Directory which will hold MSMQ’s public information.

Keep in mind that clustered MSMQ inherits the installation settings from MSMQ on the nodes, so only select the features that you need in order to minimize overhead.

Failover over the cluster is equivalent to restarting the MSMQ service.  In short, express messages will be lost, so if you need to preserve your data, then ensure that the message are either set to “recoverable” or use transactional queues.

I hope that you found this brief explanation of clustering MSMQ to be useful, and please don’t hesitate to post any comments, questions, or suggestions.

Posted by | Filed Under Windows, Networking 

Bash Tips

November 27, 2007 | Leave a Comment

1. Lost bash history

If you have a terminal open and are typing commands, then open another one and use that for a while, the new terminal won’t remember any of the commands typed in the first one. In addition, closing the first terminal, and then the second will overwrite any of the commands typed in the first terminal. Doubly annoying!

This happens because the bash history is only saved when you close the terminal, not after each command. To fix it:

Edit your .bashrc (for beginners, any file starting with a . is hidden - they contain user preferences.)
nano ~/.bashrc
No need for a sudo here Ubuntuers, this is your own file, not a system setting. I like nano, but it’s up to you, choose gedit, kate, mousepad, vi or emacs as you wish.
add the lines
shopt -s histappend
PROMPT_COMMAND=’history -a’

And save. (control - O to write out. ^ means control in nano and other software, so the bottom of the editor does actually make sense to beginners! ^X to exit.)

This makes bash append history instead of overwriting it, and makes it so that each time the prompt is shown it writes out all the history.

2. Stupid spelling mistakes

Add

shopt -s cdspell

to your .bashrc again. This will make sure that spelling mistakes such as ect instead of etc are ignored.

3. Duplicate entries in bash history

I often type cd .. multiple times in a row, when I then press UP to go back to earlier commands I don’t want to be reminded of my earlier inelegant excursions around the file system.

Add

export HISTCONTROL="ignoredups"

to .bashrc again.

Even better, add

export HISTIGNORE="&:ls:[bf]g:exit"

This will ignore duplicates, as well as ls, bg, fg and exit as well, making for a cleaner bash history.

4. Multiple line commands split up in history

Add

shopt -s cmdhist

to .bashrc, this will change multiple line commands into single lines for easy editing.

5. A couple of neat extras suggested by commenters

Press control R in bash, then start typing and you can search through your past commands much easier than just pressing UP 300 times…

Alternatively, use

history | grep "foo"

to search through your history - “foo” is the thing you are searching for. (Thanks to Ally)

cd -

goes to the last directory you were in - useful if you want to go somewhere to change something, then need to quickly flip back again.

Pressing Esc . brings up the last object you referred to. For instance, if you had just typed cat /etc/apt/sources.list , then typing rm then pressing esc . would auto complete to rm /etc/apt/sources.list

 

6.For getting the last object referred to, another way to do (that’s slightly quicker/easier to type than ESC + .) is:

Alt + . (I find it a lot easier to hit alt + . as to get escape requires moving hands quite a bit from the home row)

By far the coolest bash trick, (that I only learnt about last year) is repeating multiple parts of the commant with curly brackets {}. Ie, if you are in home and need to quickly make a backup copy of /etc/apache2/httpd.conf (to pick a file at random), instead of typing
cp /etc/apache2/httpd.conf /etc/apache2/httpd.conf.bak
you can type
cp /etc/apache2/httpd.conf{,.bak}

I use that trick a *lot*.

 

7.when trying to track down a file(s) that are hogging up disk space, I use:


for each in `ls -1`;do du -sh $each;done
528M cache
39M lib
8.0K local
32K lock
262M log
16K lost+found
4.0K mail

and then cd and follow the big directories to the big files

Posted by | Filed Under Unix, Networking 

Next Page »