Using the SQL Agent to Automate SSIS Packages

In our last post, we learned how to use SSIS to export data to a flat file. While that is certainly a step in the right direction towards removing all manual effort from our reporting processes, we still had to manually kick off that SSIS package. In this post, we are going to learn how we can use the SQL Agent to schedule this package to automatically execute.

What is the SQL Agent

The SQL Agent is a Windows service that executes scheduled tasks or jobs. Creating a SQL Agent job allows us to schedule when an SSIS package, for example, is executed automatically. SQL Agent jobs can be created to execute a number of tasks including: executing T-SQL code, SSIS packages, SSAS queries, PowerShell scripts, and a number of other types of tasks. For our purposes, we will be creating SQL Agent jobs to automatically execute our SSIS packages. ETL jobs are often automatically executed at a set time (usually nightly) via SQL Agent jobs, therefore understanding the basics of the SQL Agent are critical to becoming well-versed in the data warehouse/business intelligence toolkit contained within Microsoft SQL Server.

Creating the SQL Agent Job

Before we create the SQL Agent job, open up the “Credit Union SSIS Solution” in SSDT if it is not currently open. In the “Solution Explorer”, right click on the “Package.dtsx” package name and click “Rename”. Rename the package “Export to Flat File.dtsx”. Press the “Save All” icon on the toolbar and then exit SSDT (SQL Server Data Tools).

NOTE: the next steps require you have the proper authorization and permissions to create, schedule, and execute SQL Agent jobs. You may need to speak with your IT department for proper authentication depending on your SQL environment.

Open SQL Server Management Studio (SSMS) and login with your Windows credentials. At the bottom of the “Object Explorer” you should see an item named “SQL Server Agent”. Click the toggle button to the left of it to expand the folder. Once expanded, expand the “Jobs” folder.

This where the SQL Agent jobs are stored. Notice that two jobs currently appear – these are generated by the system and can be ignored. To create a job, right click on the “Jobs” folder and click “New Job”.

Give this job the name “Exporting to Flat File”. Leave the other default options unmodified. In every SQL Agent Job you can have many steps. So, for example, we could have multiple SSIS packages executed in the same job. This is a nice feature if you have recurring processes that need to be executed on the same schedule. You could create a single job for all of your weekly recurring reports, so that you would only need to monitor and maintain a single job instead of one for each SSIS package.

Click on the “Steps” option on the left side of the “New Job” window.

Click the “New” button near the bottom of the window to create a new job step. Name this step “Export to Flat File SSIS Package”. In the “Type” dropdown, select “SQL Server Integration Services Package”. You will notice the bottom half of the window will change significantly once you make the selection.

In the “Package Source” dropdown, choose “File system”. This is telling the SQL Agent that we will be choosing a package located in a folder instead of being stored somewhere else (there are other package storage mechanisms within SQL Server that could be selected). With “File system” selected, click the ellipsis button to the right of the “Package:” text box near the bottom of the window. Locate the SSIS package “Export to Flat File.dtsx” that we created in the previous post and renamed earlier in this post. Press “Open” once you have found this item.

After making the necessary changes, your window should look similar to the image above. Press “OK” once you have made the modifications. Once you press “OK”, you will be brought back to the “New Job” window which should now look like:

We have created the SQL Agent job with the appropriate step created to execute the SSIS package we created in the last post. Now we must schedule the job to execute at a specific time.

Click the “Schedules” option from the panel on the left. Once the window changes, press “New” towards the bottom of the screen.

When the “New Job Schedule” window appears, type “Sample Ad Hoc Execution” in the “Name” text box. Then, in the “Schedule type:” dropdown, choose “One time” from the menu.

Schedule the job to execute about ten minutes from when you are completing this step. So, I am completing this step at around 11:50 AM on 6/29/2015, so I will set the job to execute at 12:00 PM.

Press “OK” once you have made the schedule updates. Press “OK” at the bottom of the “New Job” window to save the SQL Agent job we just created. If there were any errors or you do not have the necessary permissions to create the job, SQL will throw an error and a pop-up window will appear.

You will notice that the “Exporting to Flat File” job now appears under the “Jobs” folder in the “Object Explorer”.

Once the scheduled execution time passes, right click on the job in the “Object Explorer” and click “View History”. You will see a successful execution logged with the appropriate details on the screen. If anything failed in the job, this window will tell you why the error occurred. Using this log viewer is incredibly helpful when debugging a failed SQL Agent job or step.

You can navigate to wherever you saved the exported flat file and see that the file was modified and created right at the time we specified.

And just like that you created your own SQL Agent job and automated the execution of an SSIS package. This is a critical step towards leveraging the most valuable tools within Microsoft SQL Server and its business intelligence operations. If you need to automatically import data into a SQL table or automatically execute a set of SSIS packages, using a SQL Agent job provides you with the flexibility and automation you need to be efficient and successful!