Noise Words have been Silenced

SQL Server 2008 introduces STOPLISTS as a replacement to the infamous Noise Word files. So, how easy is it to have your very own STOPLIST? Well, as it turns out very easy indeed. Although there are no tools as such for this, the T-SQL commands are very easy to use.

So, first things first – I like to create my own company STOPLIST (the semi colon is required at the end):

CREATE FULLTEXT STOPLIST MYSTOPLIST FROM SYSTEM STOPLIST;

Lets just make sure our STOPLIST has been created. We can do this simply by:

SELECT * FROM sys.fulltext_stoplists 

What has that done for us then? Well, we created a new STOPLIST from the SYSTEM STOPLIST. So all the default STOPWORDS are included automatically. Er, Whats a STOPWORD? Well a STOPLIST contains one or more words that are called STOPWORDS. Each STOPWORD is associated with a LANGUAGE, so you can have a single STOPLIST which has all your STOPWORDS for each LANGUAGE you support. You might consider not basing your own STOPLIST on the system one as it will include ALL LANGUAGE’s STOPWORDS and on our test system here that means over 14K STOPWORDS.    

If you want to see a list of the STOPWORDS for your STOPLIST, you can run something like this (replace the STOPLIST_ID with your own and change the LANGUAGE if needed).

 

SELECT * FROM sys.fulltext_stopwords
WHERE
stoplist_id = 5
AND
language_id = 2057

So, how do I add or remove STOPWORDS from my own STOPLIST? Again, this is really simple.

This script adds the STOPWORD ‘job’ to the STOPLIST we created earlier. To confirm the STOPWORD was added, you can run the previous query. Note that the language is also specified.

ALTER FULLTEXT STOPLIST  MYSTOPLIST
ADD ‘job’ LANGUAGE 2057;

To remove a STOPWORD , you would do this:

ALTER

FULLTEXT STOPLIST MYSTOPLIST
DROP ‘b’ LANGUAGE 2057;

 

Now the only thing to say is that you need to specify your STOPLIST when you create your FULL TEXT INDEX. As its good practice to DROP and CREATE your Full Text Indexes in 2008 (if upgraded or restored from a 2005 database), this is not a problem either. I really like STOPLISTS, its another one of the simple enhancements in 2008 that although not a life changing feature it does make accessing another aspect of Full Text a lot easier.

 

Database Snapshots: Quick as a Flash

The advent of SQL 2005 gave us the power of Database Snaphots. What are Snapshots? They’re just great! Someone with a lot more time than me has already written a detailed blog post on the subject: http://www.simple-talk.com/sql/database-administration/sql-server-2005-snapshots/. Unfortunately Management Studio did not come with GUI support for them, which is a real shame considering how often I use them.

However, I have written a script which I keep as a SQL Server Template (easily accessible via Template Explorer). Maybe it’s useful to you also..?

/*
Description:           Script to create a Database Snapshot of the current database
Author:                                   Nick Allan
Date:                                       Mar 2008
*/

DECLARE @Reference nvarchar(10)

SET @Reference = ‘FBCXXXX’

DECLARE @SQL nvarchar(max), @SnapshotSuffix nvarchar(100) ,@SnapshotName sysname

SELECT @SnapshotSuffix = ‘_Snapshot_’
+ IsNull(@Reference + ‘_’, ”)
+ Substring(suser_name(), CharIndex(‘\’, suser_name()) + 1, Len(suser_name())),
@SnapshotName = DB_Name() + @SnapshotSuffix

IF DB_ID(@SnapshotName) IS NOT NULL

BEGIN
PRINT ‘–Database already exists…     DROP DATABASE ‘ + DB_Name() + @SnapshotSuffix
RETURN
END

 SELECT @SQL = ‘
CREATE DATABASE ‘ + @SnapshotName + ‘
ON (
NAME = ‘ + [name] + ‘,
FILENAME = ”’ + LEFT(physical_name, Len(physical_name) - 4) + @SnapshotSuffix + ‘.ss”
)
AS SNAPSHOT OF ‘ + DB_Name()

  FROM sys.master_files
WHERE database_id = DB_ID()   AND type = 0 –Data

 BEGIN TRY
EXEC (@SQL)
PRINT ‘Created DB snapshot: ‘ + @SnapshotName
END TRY

BEGIN CATCH
DECLARE @ErrorMessage nvarchar(max),  @ErrorSeverity int
SELECT @ErrorMessage = ERROR_MESSAGE(),  @ErrorSeverity = ERROR_SEVERITY()

  RAISERROR ( @ErrorMessage,@ErrorSeverity,1 –Error State
)

PRINT @SQL
END CATCH

Properly Cleaning Up Dropped Subscriptions

If you are working in an environment utilising replication, you may find orphaned subscriptions show up on subscribers long after they have been dropped at the publisher. Well, why they aren’t cleared up when one drops a subscription or subsequently the publication itself.. you may ask… that’s a different question all together.

You can run a not so widely known command at the subscriber to clear out any orphaned subscriptions still listed on the subscribers. In fact, should I dare suggest it would actually be a great idea to implement this within your drop subscription process.

The command you can use is “sp_subscription_cleanup”. It has the following syntax:

sp_subscription_cleanup

[ @publisher = ] ‘publisher‘ ,

[ @publisher_db = ] ‘publisher_db‘

[ , [ @publication = ] ‘publication‘]
where,

@publisher => is the name of the Publisher

@publisher_db => is the name of the Publisher database

@publication => is the name of the publication, with a default of NULL. If NULL, subscriptions using a shared agent publication in the publishing database will be deleted

At the Subscriber on the subscription database, execute sp_subscription_cleanup to remove replication metadata in the subscription database.

CTE’s…What are they and why would I ever want to use one?

CTE’s were a new introduction in SQL 2005 and even now we’re looking down the throat on SQL 2008 there are still many people who are unclear as to what CTE’s are, how they would use them and when.

So what are they?

CTE is short for Common Table Expression and is basically and expression that returns a temporary result set inside of another statement. In this way they are very similar to using a declared temporary table which you then reference. They are part of the ANSI SQL 99 standard and they are particularly useful for doing recursive queries.

A recursive query is one which can be said to call itself many times over. An example would be a query which allows you to find the Manager and staff of every employee in a company. In the Adventure Works database the Employee table holds details for each employee and the ManagerId of the staff member they report to. These managerid are self joined to the respective employeeid’s in the same table. A CTE to find the information could look something like:

USE AdventureWorks ;
GO

WITH DirectReports(LoginID, ManagerID, EmployeeID) AS
(

SELECT LoginID, ManagerID, EmployeeID
FROM HumanResources.Employee
WHERE ManagerID IS NULL

UNION ALL

SELECT e.LoginID, e.ManagerID, e.EmployeeID
FROM HumanResources.Employee e
INNER JOIN DirectReports d
ON e.ManagerID = d.EmployeeID
)

SELECT *
FROM DirectReports ;
GO

This gives us the login ID, Manager Id and the EmployeeID for each user.

This statement can be broken down into three parts. The first is the ‘Root’ or ‘Anchor Member’ :

SELECT LoginID, ManagerID, EmployeeID
FROM HumanResources.Employee
WHERE ManagerID IS NULL

This tells us our top, starting level in the hierarchy.

The second part is what makes the CTE recursive as it references itself using the join : INNER JOIN DirectReports d
ON e.ManagerID = d.EmployeeID

The statements are linked together by a UNION

UNION ALL

SELECT e.LoginID, e.ManagerID, e.EmployeeID
FROM HumanResources.Employee e
INNER JOIN DirectReports d
ON e.ManagerID = d.EmployeeID

Finally we want to see our result set and this is done using the select statement:

SELECT *
FROM DirectReports ;

 

It is important to remember that the select statement must follow the CTE syntax immediatley. Try running the following as an example of how this fails when not called immediately after the CTE:

 

USE AdventureWorks ;
GO

WITH DirectReports(LoginID, ManagerID, EmployeeID) AS
(
SELECT LoginID, ManagerID, EmployeeID
FROM HumanResources.Employee
WHERE ManagerID IS NULL

UNION ALL

SELECT e.LoginID, e.ManagerID, e.EmployeeID
FROM HumanResources.Employee e
INNER JOIN DirectReports d
ON e.ManagerID = d.EmployeeID
)

 

SELECT * FROM HumanResources.Employee

SELECT *
FROM DirectReports ;
GO

The following error is thrown:

 

Msg 208, Level 16, State 1, Line 15
Invalid object name ‘DirectReports’.

 

CTE’s are just another weapon in the developers arsenal. There will be times when it is appropriate to use them and times when there is a better solution. I recently read a great article by Peter He about the performance implications of using CTE against general looping constructs. Its worth a read and can be found here.

 

SQL 2008 MERGE Functionality

One of the new SQL 2008 T-SQL features is MERGE.

The great thing about MERGE is it allows you to perform simultaneous UPDATE, INSERT and or DELETE operations on one table. Its supported by new physical operators that combine these operations so that they can be performed in a single scan rather than multiple scans.

This means that for the first time you can assign the contents of one table or query to another in a single operation

Here’s a small example which uses a stored proc:

/********Lets create the table object*******/

IF OBJECT_ID(‘TeamMember’) IS NOT NULL

BEGIN
DROP TABLE TeamMember
END

GO

CREATE TABLE TeamMember

(

EmpId INT PRIMARY KEY IDENTITY (1,1),

FirstName VARCHAR(200),

LastName VARCHAR(200),

Department VARCHAR(200)

)

GO

/********Insert some Team Members*******/
/********here using the new insert multi row functionality available in 2008*******/

INSERT INTO TeamMember (FirstName,LastName,Department)
VALUES

(‘James’,‘Hill’,‘IT’),

(‘Sarah’,‘Parker’,‘Sales’),

(‘Nick’,‘Thompson’,‘Marketing’)

GO

/********Create a procedure that might do some Team Member admin in ‘the REAL world’ !*******/

IF OBJECT_ID(‘up_InsertTeamMember’) IS NOT NULL
BEGIN
DROP PROCEDURE up_InsertTeamMember
END

GO

CREATE PROCEDURE up_InsertTeamMember
(

@FirstName VARCHAR(200),
@LastName VARCHAR(200),
@Department VARCHAR(200),
@EmpId INT OUTPUT

)

AS

BEGIN

/********Lets do the MERGE here*******/
MERGE TeamMember
USING (SELECT @FirstName,@LastName)
tm (FirstName,LastName)
ON TeamMember.FirstName = @FirstName AND
TeamMember.LastName = @LastName

/********If its not there lets create a new Team Member*******/

WHEN NOT MATCHED THEN
INSERT (FirstName,LastName,Department)
VALUES (@FirstName,@LastName,@Department)

/********If it is there lets just set the existing id to the ouput variable*******/

WHEN MATCHED THEN
UPDATE SET @EmpId=EmpId,Department = @Department;

/********Grab the new ID*******/
SET @EmpID = COALESCE(SCOPE_IDENTITY(),@EmpID);

END

RETURN;

go

/********Heres the procedure, try adding a new memberand then a team member with the same FirstName and LastName but different department*******/

DECLARE @EmpId INT,@Message VARCHAR(100)

EXEC up_InsertTeamMember @FirstName = ‘James’,@LastName = ‘Smith’,@Department = ‘Sales’,@EmpId = @EmpId OUTPUT

PRINT @EmpId

GO

SELECT * FROM TeamMember

GO

 

Insert multiple rows

How many times have we had to write multiple INSERT INTO statements? Well, SQL Server 2008 comes to the rescue. Its a small feature but still very useful.

You can now insert multiple rows into a table in a single statement.

INSERT INTO T1 (Col1,Col2,Col3)
VALUES

(‘Row1′, ‘Value’,‘Test’),
(‘Row2′, ‘Value’,‘Test’),
(‘Row3′, ‘Value’,‘Test’)

This inserts 3 rows into the T1 table. Certainly time-saving and makes those sometimes hefty insert scripts smaller.

SQL Bits II

Last Saturday (1st March 2008) was the second SQL Bits meet, this time in Birmingham.  Once again it turned out to be a great event with interesting speakers and all for the grand price of…zilch!!

So I would personally like to extend my thanks for everyone that helped put it together and I look forward to the next one.

I notice that Chris Webb has set up a SQL Bits Facebook group so for those on FB, check that out.

Activation Procedure Being Disabled

Today I had a situation where Service Broker appeared to be running successfully but the activation procedure was not being called and no receive processes were being seen in Profiler. It turned out that the activation procedure had ‘switched off’ although there was nothing I could see that indicated this other than the lack of activity. The process was restarted by simpling calling the activation procedure from MS.

The problem in my scenario started when the broker attempted to run a CLR function on a box on which CLR was not enabled (doh!) and by the time this had been enabled and the queue had grown significantly, the activation procedure had shut down.

Service Broker Deployment Numero Uno!

We have just deployed my first Service Broker project to the live environment and touch wood everything appears to have gone ok. Sure it took a little massaging to get it through and we certainly learned a few things along the way but thats all part of it right.

Working for a well known job board we are using service broker in conjunction with SQL CLR to do asynnchronous string manipulations to job adverts.  Orignally this was something we did through replication but it became clear very quickly that fast as CLR is, replication latency would be effected.  So instead we had a re-think and moved the process offline.  We’re sure that there are many other applications for Service Broker within our business now.

Service Broker Terminology (A Quick Guide)

Microsoft’s new Service Broker feature in SQL 2005 introduces a number of new concepts, processes and terminology which for some may make the idea of using Service Broker a little daunting. On this site we hope to ease some of that pain by giving our thoughts, examples and explanations on the platform.

To start with lets have a look at some of the terminology which may be new to many.

A:

Asynchronous

Asynchronous in process terms means a task which can continue independently “in the background”. Other processes may be started before the asynchronous process has finished.

In Service Broker this means that we can effectively decouple messages from the processes that initiated them. A sending process can send a message and then continue to completion or start a number of other processes without ever having to know the status of the sent message. This could be particularly useful if you use multiple systems that may not function at the same speed. The faster initiating system could start a process in a ‘fire and forget’ manner allowing the slower system to process at its own speed.

C:

Contracts

A contract is an agreement between two endpoints within a Service Broker conversation. It determines what type of messages can be sent and who can send them. To preserve its integrity, once a contract is defined its list of message typ0e cannot be changed.

Conversation

A conversation in the context of Service Broker is defined as a reliable, ordered and asynchronous transfer of messages between conversation endpoints. Conversations can be either dialog or monolog, however in 2005, monolog conversations were taken out, but will likely be included in a future release. Service Broker will continue trying to deliver a message irrespective of database or network failure hence reliability. In terms of ordering, Service Broker will ensure that messages sent on the same conversation will be delivered and processed in the order in which they were sent. This could be particularly important in multi-threaded environments or where payload size or complexity exists.

D:

Dialogs

A dialog conversation, or dialog, is a conversation between two services

The lifetime of a dialog lasts from the time that the local SQL Server instance creates the dialog until an application either explicitly ends the dialog or receives an error message associated with the dialog. Each participant is responsible for explicitly ending the conversation when the application receives a message that indicates an error or the end of the conversation. In most services, one participant is responsible for indicating that the conversation is complete and successful by ending the conversation without an error. Whether this is done by the target or the initiator depends on the purpose of the conversation.

E:

Endpoints

Endpoints are used to allow Service Broker to send and receive messages across a network. Out of the box an installed instance of SQL Server will not contain any Service Broker Endpoints for security reasons. In order to use Service Broker you must first create an endpoint.

G:

Groups (as in conversation groups)

Every conversation is associated with a conversation group. When a message is sent or received the conversation group is locked to stop any other messages can be sent on it until the transaction holding the lock completes. This ensures that even with many queue readers active a conversation can only be accessed by one queue reader at any one time.

M:

Messages

Messages are as you might expect, the payload or information that is being exchanged during a conversation. This information is divided into two parts, one being the message header and the other being the message itself.

The header contains information about he message such as the message type. This is a kind of descriptor for the message that is being relayed.

The message body is a VARBINARY(MAX) datatype which can contain up to 2GB or data. This data can be of any SQL datatype that can be cast to VARBINARY(MAX)

Q:

Queues

Queues are one of the ways that Service Broker manages the work being transmitted, by storing it in the database. In the same way as other queues manage data or objects it allows items to be stacked up in order by an initiating process which can then move on to another task. Many people have probably used some kind of process table in sql server to manage data, one process puts it on and another pulls it off. This is the same idea.

These queue tables are hidden system tables but there are system views that will allow you to read the data.