Dec 4, 2018

Background Intelligent Transfer Service - BITS

BITS or Background Intelligent Transfer Service has been available for many years with Windows Servers, it is generally considered only as a background service for Web Servers, Application, etc., though it can be used by System Administrator, IT Administrators or DBAs too. As a Database Professional I find it very useful thus I would like to share a demo of this service. The purpose of BITS is simple, transfer files between source and destination with fault tolerance capabilities.
Making it easy to use, BITS cmdlets are available with PowerShell too. The purpose of BITS as written on Microsoft Documentation gives more insight on its capabilities:
Background Intelligent Transfer Service (BITS) is used by programmers and system administrators to download files from or upload files to HTTP web servers and SMB file shares. BITS will take the cost of the transfer into consideration, as well as the network usage so that the user's foreground work has as little impact as possible. BITS also handles network interruptions, pausing and automatically resuming transfers, even after a reboot. BITS includes PowerShell cmdlets for creating and managing transfers as well as the BitsAdmin command-line utility.
Here is quick guide on using BITS with PowerShell:

Importing Module in PowerShell
  • This will be a good practice to run PowerShell with elevated rights (Right Click and Run as administrator) or run in same security context each time, otherwise there may be inconsistencies as some of the resources may not be available.
  • Once Powershell Opens, execute Import-Module BitsTransfer
  • You should not get any message or prompt if Import was successful
image
  • This demonstration has been tested with PowerShell Version 4 and 5. You may verify the version by executing $psversiontable.psversion from PS prompt.
Pre-Requisites before starting transfer
  • Obviously you would need to know the source and destination locations and file to be transfer. Also make sure both locations are accessible from PS prompt.
  • BITS provides two types of transfer modes, Synchronous and Asynchronous.
    • In Asynchronous mode BITS job will be registered and pointer will be returned to prompt, job will continue in background by service.
    • In Synchronous mode, BITS job will be registered and pointer will not be returned to prompt unless transfer completes.
Limitation
  • The only major limitation with BITS is that it can only be run if user’s session is active, i.e. BITS commands cannot be scheduled by schedulers such as SQL Agent, Task Scheduler, etc., with only exception that user can disconnect the session without logging off from the Windows Session and the process will continue. If this restriction is removed then this will become my favorite file transfer utility.
Cmdlets
  • To start\initiate transfer: start-BitsTransfer –Source [sourcepath] –Destination [DestinationLocation] –Asynchronous [If not specified then it becomes Synchronous]
  • To pause transfer: suspend-BitsTransfer 
  • To resume transfer: resume-BitsTransfer
  • To modify existing transfer: set-BitsTransfer
  • To get existing transfer jobs: get-BitsTransfer
  • To stop\terminate transfer: remove-BitsTransfer
  • To complete the transfer job: complete-BitsTransfer
Demonstration
  • To demonstrate, I will transfer a 800 GB file using BITS as shown in the image below and will use some of the commonly used parameters as well as will get more useful details out of the job.
image

  • The source and destination are “C:\demo\source\SSMS-Setup-ENU.exe” and “\\swarns\demo\destination\” respectively

Starting Transfer in Asynchronous mode
  • import-module bitstransfer
  • start-bitsTransfer -Source "C:\demo\source\SSMS-Setup-ENU.exe" –Destination “\\swarns\demo\destination” –Asynchronous
image
  • The image above shows the job has been registered and current job status as Connecting

Suspending the job
  • I have intentionally suspended this job so that I can read properties of this job before copy finishes, to do that the cmdlet is suspend-bitstransfer
  • To suspend you need to provide jobid as a parameter in two different ways, first is using PIPE and other is using variable. I am going to use PIPE and first need jobid
    • get-bitstransfer | select jobid
image
    • get-BitsTransfer -jobid 1c6eaea9-efb5-4e1c-bae9-64b79b5d8433 | suspend-BitsTransfer

Reading BITS properties in easily readable format
  • I have intentionally suspended this job so that I can read properties of this job before copy finishes, to do that the cmdlet is suspend-bitstransfer
  • get-BitsTransfer -jobid 1c6eaea9-efb5-4e1c-bae9-64b79b5d8433 | select *
image
  • The image above shows basic output of command and this can be formatted to get more usable
  • I have used the following scriptIf you want to test the following script, please write the following scriptlet in single line, I have intentionally divided the scriptlet at each Pipe to make it readable
    • Get-BitsTransfer -jobid  1c6eaea9-efb5-4e1c-bae9-64b79b5d8433 
    • | SELECT *,  @{name='Xferred_percent';expr={$_.BytesTransferred/$_.BytesTotal*100}}, @{name='Minutes'; expr={((Get-Date)-$_.creationtime).TotalMinutes}}
    • |SELECT @{name='BytesPerMinute';expr={$_.BytesTransferred/$_.Minutes}},  @{name='EtaInMinutes';expr={$_.Minutes/$_.BytesTransferred*($_.BytesTotal-$_.BytesTransferred)}}, *
    • | SELECT @{name='CompletionEstimate';expr={(Get-Date).AddMinutes($_.EtaInMinutes)}}, Xferred_percent, Minutes, BytesPerMinute, JobState | Format-Table
image
  • The script above may sound little bit complex, but let me simplify it below:
    • A PIPE  “|” is used to pass the value to next cmdlet or expression, to define an expression, a hashtable “@{}” is used with name and expr tags. Calculations can be performed in expr tag. Next PIPE can access the expr by refering to name defined in hashtable.
image
Last Step
  • To resume the suspended job, the cmdlet will be get-BitsTransfer -jobid 1c6eaea9-efb5-4e1c-bae9-64b79b5d8433  | resume-BitsTransfer –Asynchronous #Optional
  • Once transfer completes, if you started job with Asynchronous file transfer, this would require you to manually complete the job by issuing get-BitsTransfer -jobid 1c6eaea9-efb5-4e1c-bae9-64b79b5d8433  | remove-BitsTransfer
image

Finally
This service is robust, runs very well, I never got into any issues after transferring bunch of files. I have used robocopy too for many transfers, but could not read transfer statistics asynchronously. At the same time the inability to use BITS if user did not login, reduces its usage for many. I wish if there was no such restriction. Thanks for reading this article, please leave your comments if you find it useful.








Nov 5, 2018

Git and GitHub for Version Control on Windows

Git is Version Control System also known as VCS that you install on your local machine and use it to check in, check out, commit your code just like any other VCS like TFS. However, in case of multiple developers working together can upload or merge their changes in centralized location called GitHub (also, can have in house Git Server).
Here is quick guide on using Git with GitHub.


Getting Git for Windows
image

  • Go through the installation process after downloading

image

  • By default Git Bash and GUI will be integrated with Windows Explorer, but can be disabled while installing.
image

  • You may keep default options during installation process, default options would work for most of users.

Setting up GitHub Account and Settings
  • Open https://github.com/ and create your account or Login if you already have account.
  • You have to add a project on GitHub where you would be creating repositories, each project can have multiple repositories and branches
image
  • Click on create repository to add and commit repository immediately.
Adding files to repository
  • You may directly add files and folders to GitHub using Web Portal GUI or by using CLI also known as Git Bash.
  • Git bash was installed as part of Git installation.
Adding files from Web Portal GUI
  • Adding files from Web Portal is very simple, you may simply Upload or Create New file.
  • Also, note that using Git Hub does not require you to install Git for Windows as it is completely web based.
  • You would get option to Upload or Create New file as soon as you go into Repository as shown below:
image

  • Once you select the file(s) as shown above, the file(s) should be committed
image
  • Add comments to commit and press Commit Changes to apply changes, this will complete the check in process using URL or Web Portal of GitHub.
Adding files using Git Bash

This process first requires you to create Clone of Repository from GitHub to local and then make your changes in local repository, once done then you can upload your changes to GitHub using Git Bash. Step by step process is given below.
  • Open Git Bash program. (Git Bash accepts linux like commands, for example: cd, pwd, ls, etc.)
      • If you would like to change current directory to C drive then command will be “cd /c”
      • Same way if you would like to see content of current directory then command is “ls”
      • To get current directory (aka Working Directory) path the command “pwd”
  • The first step requires to clone the repository from GitHub.
  • In this demonstration the directory used to keep clone is c:\github hence the command to change directory will be “cd /c/github”.
image
  • Also you would require git URL of your repository. You may get git URL from github repository page, by selecting option Clone or download as per screenshot below
image


  • Make sure that you are in working directory where you want to clone the Repository and execute git clone command “git clone [git URL]”
image

  • Git will use your current identity (Login name or ID) against changes and activities you would perform, you may want to change it to your preferred user name using config command.
  • If you add global switch to config command the changes will be applied to all repositories, otherwise changes will be applied to current repository only.
image
  • Global config below
image
  • Before you could make changes like adding files etc. you will have to checkout the repository. In previous steps you cloned the repository.
  • Command to checkout repository is “git checkout master” (master is main copy and you can create branches from master, for example development or test branch)
image
  • Now you can add files to the working directory or create new, once done add file(s) to repository “git add [filename]”
image
  • The next step is to commit changes in local repository using command “git commit –m “comments”
image
  • The final step in the process is to push the changes and the command is “git push”
image
  • To validate, you may go to Git Hub URL and you should see latest changes you made from Git Bash.
image












Optimizing Indexes with Execution Plans

Contact Me

Name

Email *

Message *