Blog is moved!

The blog has been moved to http://www.biztalkadmin.com

How to do a good health check part 2

In my first post you could read about the health check around network and updates provided by Microsoft and other manufactures, IBM, HP Dell and so on.

This part is more related to how to health check BizTalk itself. How to set it up correctly.

The Host separation policy

BizTalk solutions typically consist of executable artifacts - receive locations, pipelines, orchestrations, send ports, and so on. BizTalk hosts provides a way to logically group these artifacts and allow you to instance each host on one or more BizTalk Servers. A Host instance represents a physical run time windows service or process running on a BizTalk. This gives you a very flexible solution to host BizTalk application components, there are many factors which should be taken into account when distributing your applications across hosts.

-Performance

• When you create a new host a set of dedicated databases tables is provisioned in the BizTalk databases. In the BizTalkMsgBoxDB they act as the work queues for that host. This can help to gain low latency situations by creating multiple queues for different workloads, thus presenting messages with low latency requirements getting delayed behind large batches of none "time-critical messages".


• Each host instance has its own set of system resources such as memory threads and handles in the .NET thread pool. Instead of hosting all executable artifacts in a single host i can be beneficial to spread the load across multiple hosts.


• BizTalk throttling is implemented on the host level. You may experience that you need to set a specific throttling parameter in order to control individual artifacts. If you need to do this I would recommend you to separate artifacts across different hosts so that you don't apply unnecessary throttling to all artifacts.

- Functionality

• Its recommended to separate reviving, orchestration, sending and tracking functionality into separate hosts. This will help you to manage and scale each type of processing independently. IF you want to stop all reviving or all orchestration processing. Or adding additional servers to the BizTalk group to handle dedicated orchestration processing only.


• You should also have hosts for application that only can run on one server, POP3, FTP, MSMQ receive. These hosts should be grouped and clustered in the BizTalk console· All others can remain in hosts which supports multiple running host instances.


• Separation application processing across hosts ultimately leads to a multiple runtime processes hosting you applications. As BizTalk can host custom written components (all custom components you have), host separation can be used to provide process isolation and protection. If you experience instability in any custom components, you should isolate them into their own BizTalk host. This should be done to prevent the instability to affect other applications.

- Security

• Each BizTalk host is assigned to a windows group which is used to give the host permission to specific tables in the BizTalk database. This is important when dealing with highly sensitive data.


• Certificates can be utilized in BizTalk for message signing and encryption. This settings is assigned in the host level and therefore you may need to separate hosts in order to represent different identities.


• All host instances are assigned wit ha service account to run under. Access to external systems is often a requirement for this account in order to process outbound messages through the adapters.

Set up a dedicated tracking host
Any hosts that hosts tracking is responsible for moving the DTA and BAM tracking data from the MessageBox database to the BizTalk Tracking (DTA) and BAM Primary Import databases. The moment of tracking data has an impact on the performances of other BizTalk artifacts running on the same host that is hosting tracking. Therefor you should have a dedicated host that does nothing but tracking.

To use a dedicated host for tracking also allows you to stop other hosts without interfering with the BizTalk server tracking. The movement of tracking data out of the MessageBox database is critical for healthy systems. IF the BizTalk host is responsible for moving the tracking data in the BizTalk group is stopped, the Tracking data decode service will not run. Impacts of this is:

• HAT tracking data will not be moved from the MessageBox to the BizTalk Tracking database. Same goes for BAM tracking data, this will not be moved to BAM Primary Import database.

• Because data is not moved, it cannot be deleted from the BizTalkMsgBoxDb.


• When the tracking data decode service is stopped, tracking interceptors will still write tracking data to the MessageBox database. IF the data is not moved, this will cause the MessageBox database to become enormous. which will impact performance over time.


• It is also important for security to have a dedicated tracking host, since the tracking host has access to all the messages across the message box database.


• Turn of tracking on all other hosts, they will still write tracking but they wont have read access.

Isolated Adapter per Application Pool
An application pool which hosts an isolated adapter represents the physical implementation of an isolated host instance. This means that the application pool (w3wp.exe process) becomes an instance of BizTalk because the isolated adapter loads the BizTalk runtime components into the process to perform message processing. BizTalk only supports a single isolated adapter per application pool.

Remove unnecessary BizTalk items, components and settings
It is important to keep a clean environment. Remove old receive locations, send ports and applications that are no longer in use. Artifacts which only were used during testing and development should be removed. The output from the BizTalk Documenter tool can be used for this.

The BizTalk configuration file
BizTalk runtime process is like other .NET applications and have configuration files that can be used to control certain application behaviors. They control runtime validation, orchestration dehydration and provide custom application settings for custom .NET components hosted in BizTalk

BizTalk Registry Key Validation
BizTalk stores a variety of configuration data in the windows registry. This data can have an impact on the behavior of the BizTalk runtime processes and associated services. There is one specific area area which is recommended to tune for the default registry values and these control the .NET thread pool used by BizTalk. You can find the values here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$hostname\CLR Hosting

I will add more information in part 3 to 5. There is a lot to go trough and as said earlier, I don't want to spoil it all by giving away everything immediately (plus there is a lot to read).

Check part 1

How to do a good BizTalk Healthcheck part 1

As many of my friends know, my biggest interest is tuning and maintaining a BizTalk environment.

I made this list to make it easier for people to do a good health check. I would advise people to do the run at least once every 6th month. Calling Microsoft "in" to do the job is also recommended if you don't feel safe to do it yourself.. There are also many MVP's to help you out. You are also free to contact me. And i need to mention that i have been working with BizTalk 2006 R2 mostly so this guide is for BizTalk 2k6 and 2k6 R2, but can also be used for newer versions.

If you see something i dont have here, please feel free to leave me a comment.

Patching:
- What are the routines to install windows critical updates.
Good routines are critical.

-You should also run the Microsoft Baseline Security Analyzer (MBSA) on the servers Both SQL and Biztalk server. This is to identify and inconsistencies between the deployed updates and the current list of recommended updates

- Update COM+ and MSDTC.
Both COM+ and MSDTC are used by BizTalk, patches and hotfixes for these services often provide hotfixing to improve tuning and stability.

General Network
- DTCPing of all the servers
There should be no packets lost during this ping.

- Transfer 100mb data
if you have a 1gb network the response should be 5 seconds or faster (average between 3-5 seconds). Do to all the servers. If you are running a 100mb network up to 20 seconds is within the accaptable transfer rate (average is around 8 - 10 second)

- Pathping
Do a pathping from the BizTalk server to all the SQL servers ("pathping

- TCP/IP porth Echaustion
Troubleshoot to see if you are using less then 3000 ephemeral ports. TO do the counting write the following command on the BizTalk Servers "netstat -ano -p tcp" Count the number of unique Local Address TCP ports open above 1024 for each IP adresse. Using TCPView tool makes this is a lot easier.

- DBNETLIB Exceptions
Avoid this, DBnetLib (Database Network Library). The most common error when this occurs is when one of the BizTalk MessageBox becomes extremely busy. Attempts to communicate with the busy MessageBox database results in a timeout. Look for 5410 errors in the EventLog. (example of error message:
Event Type: Warning
Event Source: BizTalk Server 2006 R2
Event Category: BizTalk Server 2006 R2
Event ID: 5410
Computer:
Description:
An error occurred that requires the BizTalk service to terminate. The most common causes are the following: 1) An unexpected out of memory error. OR 2) An inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

-NetBIOS over TCP/IP
This is only a check for the BizTalk servers that have a direct connection to the internet such as those residing in a perimeter network hosting HTTP and SOAP adapters If the BizTalk servers reside in a vulnerable network location then check to ensure any internet/public facing network adapters have NetBIOS over TCP/IP disabled

Server general

-Time sync
Its vital that the time is synchronized between the BizTalk server, and SQL server. Check to see if the time is within the valid synchronization by typing the following command in command prompt "w32tm /stripchart /computer " to resynchronized the clocks type the following in the command prompt window "w32tm /resync /computer "

- BIOS Version
BIOS should be updated, this is because the releases from the manufactures can provide better satbilitet, and network related performance.

- Update the Certificate RevocationListUpdates
If it takes a long time to start up a BizTalk server it may be because you don't have access from the BizTalk server to reach the microsoft.com domain. You might get some startup issues because the .NET framework will try to download the Certificate Revocation LIST (CRL) from microsoft.com. You can update your servers manually by following these two links: http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl and
http://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl

I will come to more related to the BizTalk Server Configuration later. I dont want to spoilt it all by giving you everything at once.

Check part 2

Usefull SQL queries part 2

I've gotten a lot of request on more queries. There is another one I use often. The one to determine when during the day you have the most activity.




SELECT datepart(HH, [dtInsertionTimeStamp]) as date,
count(Convert(char(10), [dtInsertionTimeStamp], 108)) as ant FROM [BizTalkDTADb].[dbo].[dta_MessageInOutEvents] WITH (NOLOCK) WHERE [dtInsertionTimeStamp] BETWEEN convert(datetime, '01-01-2011 00:00:00', 120) AND convert(datetime, '07-08-2011 23:59:59', 120) GROUP BY datepart(hh, [dtInsertionTimeStamp])ORDER BY [date]

all you need to do is change the date after the WHERE clause.

Good luck

Useful BizTalk SQL queries

Hello, today I want to share all the SQL queries i use a lot when monitoring BizTalk. As I've mentioned before i use MessageBoxViewer alot, but sometimes i only need some small information, or just a simple query towards the Tracking.

Marklog:
Marklog is a table in the tracking database store a string every time BizTalk backups the tracking database, this is a known bug in BizTalk 2006 - 2009. And it will most probably never be fixed, make sure the marklog table isn't to big, it might give you some performance issues towards the tracking database.

SELECT [MarkName] FROM [BizTalkDTADb].[dbo].[MarkLog] WITH (NOLOCK)

See count of all messages and port name for a specified date. (as long as you have tracking on all receive and send ports.

SELECT CONVERT(char(10), [BizTalkDTADb].[dbo].[dta_MessageInOutEvents].[dtInsertionTimeStamp], 101) AS InDate, COUNT([BizTalkDTADb].[dbo].[dta_MessageInOutEvents].[nPortId]) AS numb, dta_PortName.strPortName AS name FROM [BizTalkDTADb].[dbo].[dta_MessageInOutEvents] WITH (NOLOCK) INNER JOIN [BizTalkDTADb].[dbo].[dta_PortName] WITH (NOLOCK) ON [BizTalkDTADb].[dbo].[dta_MessageInOutEvents].[nPortId] = [BizTalkDTADb].[dbo].[dta_PortName].[nPortId] WHERE [BizTalkDTADb].[dbo].[dta_MessageInOutEvents].[dtInsertionTimeStamp] BETWEEN CONVERT(datetime, '05-10-2011 00:00:00', 120) AND CONVERT(datetime, '05-17-2011 23:59:59', 120) GROUP BY CONVERT(char(10), [dta_MessageInOutEvents].[dtInsertionTimeStamp], 101), [dta_PortName].[strPortName] ORDER BY InDate, numb DESC

The last is just a simple count of all messages registered in the MessageInOutEvents.

SELECT Convert(char(10), [dtInsertionTimeStamp], 101) as date, count(Convert(char(10), [dtInsertionTimeStamp], 101)) as ant FROM [BizTalkDTADb].[dbo].[dta_MessageInOutEvents] WITH (NOLOCK) WHERE [dtInsertionTimeStamp] BETWEEN convert(datetime, '05-10-2011 00:00:00', 120) AND convert(datetime, '05-17-2011 23:59:59', 120) GROUP BY Convert(char(10), [dtInsertionTimeStamp], 101) ORDER BY [date]

Feel free to use these queries as you wish. (remember to change the date in the query).

Make sure the dates are within the time you have tracking stored, and the query execution time will vary depending on how many massages you have during a day and for how long you want to gather the information.

What is a BizTalk Administrator

This is a question i get all the time. What is a BizTalk Administrator?

A BizTalk administrator has a lot of important roles to take care of. A good BizTalk Administrator should have good knowledge of Microsoft servers, SQL, PowerShell, IIS, .NET, infrastructure, Network and preferably Oracle, fileshares, WCF and other adapters used by your company.

The biggest difference between a BizTalk developer and Administrator is that the administrator has nothing to do with developing integration (basic file integrations can be created by the administrator). The administrator should have good knowledge of BizTalk, how it works, how to run health checks, stress testing, migrating integration and know how to create good integrations. It's so sort of an Architect job, but at the same time not.

The Administrator should have the full responsibility of accepting new integrations, integrations violating the framework should not be accepted, the ability to suspend integrations giving to many errors not defined in the documentation of the integration. Or even integrations giving the environment problems or issues should be suspended.

The administrator should have full knowledge of the health of the environment and how to tune it, there is no environment the same, so a full knowledge of the environment running at the company should be well known for the administrator.

Knowledge of the adapters being used, how they work at how to check for errors on different adapters, a knowledge of the most common errors and what they mean. The administrator should also know how to set up a complete environment from scratch and how to do a disaster recovery if everything goes wrong. Knowledge on how to set up the backup jobs, and the different SQL job required for BizTalk, a good knowledge of the BizTalk SQL is also impotent.

Creating PowerShell commands to enhance the productivity of BizTalk. How to export multiple messages at the same time (BizTalk has these powershell method already made for you, all you need to do is change them for your need).

A BizTalk administrator should be well known of throttling, and how to change edit and debug possible throttling errors.

Performance Counters are many for BizTalk, which one are impotent to monitor at all times. And how do you read them, a BizTalk administrator should know this. How does a DFS fileshare work, what are known bugs and possible scenarios if you create a lot of receive locations and send ports towards one file server?

How to set up a "Microsoft best practice" environment, is this something for you or do you have to do it another way?

How many servers should you be running, what will happen in the next two years, are the integrations able to migrate to a newer BizTalk version.

How does SCOM work, how can it enhance your stability and monitoring. How does the BizTalk pack for SCOM work, and how can you use the benefits from this in your everyday work.

Should you use a third party monitoring service like IPM, Minotaur or BizTalk Monitord (I recommend the last one). Do you need tracking on some if the integrations, what are you tools for this, how do they work?

This is just a little of it all, more will come later. So keep following.

Best wishes!

Useful programs for a BizTalk Administrator

There are a couple of programs that makes the day easier for a BizTalk Administrator.

MessageBox Viewer
This is a product from Microsoft. MessageBox Viewer simply does a quick and vital health check of your environment. Generates a report and a file for Terminiator (another program from Microsofts tech guys).

Its and easy to use GUI and should be run at least once a month to make sure everything is running smoothly.

If errors occurs there's usually an explanation behind the error on how to solve it!

MessageBox Viewer can be downloaded here:
http://blogs.technet.com/b/jpierauc/archive/2007/12/18/msgboxviewer.aspx

Terminator
This is another tool to do simple queries towards the BizTalk database to clean up some of the errors discovered in MessageBox Viewer eq. Orphan messages, tracking and so on.

Terminator for BizTalk can be downloaded here:
http://blogs.msdn.com/b/biztalkcpr/archive/2009/10/01/biztalk-terminator-download-install-info.aspx

BizTalk Monitord
BizTalk Monitord is an open source monitor solution for BizTalk created by me. It gives you the information regarding the hosts, ports and receive locations. Easy to use, SMS warnings and so on. You can add many environments and users.

SMS warnings can be turned on if you use clickatell.

BizTalk Monitord can be downloaded here:
http://sourceforge.net/projects/biztalkmonitord/files/