Dec 19, 2019

AWS Lambda Functions

Create Lambda function

Go to AWS Lambda homepage and login to console.
Press "Create Function" button.
Enter function name.
Select a language (environment) from Runtime dropdown. I will use node.js 8.10.
For the first time you can create a new role for permissions. Select "Create a custom role" from Role dropdown box.
A new page will be shown. Select "Create a new IAM Role" from IMA Role dropdown and then give a name. Press "Allow" button.
This will create a new role and browser will return back to "Create function" page.
Press "Create function" button.
It will redirect you to function page.
Initially the page adds a template code which returns success code and a hello message. We will continue with this code.
 

exports.handler = async (event) => {
    // TODO implement
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

2. AWS CLI to call the function

Download AWS CLI setup to your computer. For Windows download link is here. For other operating systems you can find links at the same page.
After installation go to aws cli folder and run following command to start configuration.
aws configure
We need following info.
AWS Access Key ID [None]: "Your Access Key"
AWS Secret Access Key [None]: "Your Secret Key"
Default region name [None]: 
Default output format [None]: 
To get keys go to AWS console
Click to your name and select "My Security Credentials".
On "Access keys" section click to "Create New Access Key" button to create new keys. You have to save your secret key because it will displayed only once.
Now return back to aws cli console and enter Access Key ID and Secret Access Key.
Enter region name like "us-west-2".
Enter json for default output format.
Now you can run your functions.
We need our function name as a parameter. In lambda function page copy arn.
Run aws command to run function. The lines between "{ }" is the response from our lambda function. You can see hello message inside txt file which we passed as a parameter.
aws lambda invoke --function-name arn:aws:lambda:us-west-2:123456780315:function:FirstLambda "c:\aws.txt"

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
aws.txt file content
{"statusCode":200,"body":"\"Hello from Lambda!\""}

Sep 20, 2019

Upload file to Console Webapi (Owin)

Upload a file to a console based webapi host (owin).

Method 1: MultipartFormDataStreamProvider

 

[HttpPost]
public async Task PostFile()
{
 string root = AppDomain.CurrentDomain.BaseDirectory + "Uploads"; 

 var provider = new MultipartFormDataStreamProvider(root);
 await Request.Content.ReadAsMultipartAsync(provider);
 foreach (var file in provider.FileData)
 {
  var buffer = File.ReadAllBytes(file.LocalFileName);  
 }
 return new HttpResponseMessage() { Content = new StringContent("OK") };
}

Postman Post Sample

Select form-data for Body. Enter key name and select a file for the value.
POST /api/file/PostByteArrayAsync HTTP/1.1
Host: 127.0.0.1:5000
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Length: 15143
Connection: keep-alive
cache-control: no-cache


Content-Disposition: form-data; name="file"; filename="/C:/Users/oktay/logs.txt


------WebKitFormBoundary7MA4YWxkTrZu0gW--

Method 2: ReadAsByteArrayAsync

This method requires filename in header. Add key/value for "filename".
 

[HttpPost]
public string PostFile()
{
 string fileName = Request.Headers.GetValues("filename").ElementAt(0);

 string root = AppDomain.CurrentDomain.BaseDirectory + "Uploads\\";
 var filePath = root + fileName;

 var fc = Request.Content.ReadAsByteArrayAsync().Result;
 File.WriteAllBytes(filePath, fc);

 return "uploaded";
}

Jul 26, 2019

SQL Server Shrink

Problem 

  • SQL Server disk usage is very high. 
  • SQL Server Management Studio is very slow.

Investigation

Windows Event log:
  • Starting Up Database
  • Could Not Allocate Space for Object Name in Database ‘DB’ Because the ‘PRIMARY’ Filegroup is Full
  • CREATE DATABASE or ALTER DATABASE failed because the resulting cumulative database size would exceed your licensed limit of 10240 MB per database.

Check

Size of database files (data and log) may be at maximum limit.

Size Limitation Database Size for Microsoft SQL Server versions
  • Microsoft SQL Server 2000 Desktop - 2 GB
  • Microsoft SQL Server 2005 Express edition has a database size limit to 4GB
  • Microsoft SQL Server 2008 Express edition has a database size limit to 4GB
  • Microsoft SQL Server 2008 R2 Express edition has a database size limit to 10GB
  • Microsoft SQL Server 2012 Express edition has a database size limit to 10GB
  • Microsoft SQL Server 2014 Express edition has a database size limit to 10GB
  • Microsoft SQL Server 2016 Express edition has a database size limit to 10GB

Solution

Shrink data file and log file.

Initial sizes:

  1. Connect SQL Server via SQL Management Studio.

  2. Find database in left pane, right click and select Tasks > Shrink > Files.


  3. Select "Log" under "File type" list. Select "Reorganize pages...". You can enter minimum MB in text box as shown next to it. I entered 1 and completed shrink.


  4. Now you may also want to shrink data file. Repeat 2nd and 3rd steps for "File type: Data".
  5. I entered 1000 and completed shrink.

File sizes after shrink:


Jul 23, 2019

Nvidia GPU driver errors

After Windows10 update Nvidia GPU driver corrupted. You can find driver downloads in Nvidia website (http://www.nvidia.com/Download/index.aspx)

Setup may not continue for some reasons. 2 errors described below.

Error1: Windows cannot verify the digital signature for this file

Disable enforced driver signing Run command line and type
shutdown /f /t 0 /r
Press "F8" and continue.
Click ‘Troubleshoot’.
Click ‘Advanced Options
Click ‘Windows Startup Settings
Click Restart.

After restart select ‘Disable driver signature enforcement‘ from the list by hitting F7. Then continue to install driver after Win10 booted.

Note: The shutdown command does not work in a RDP/Termial Session, otherwise error “The parameter is incorrect.(87)” occurs.

Error2: The graphic driver could not find compatible graphics hardware || Nvidia installer not compatible with this version of windows

Exit setup.

Go to Device Manager > Display Adapters > Select device from tree > Right click and select Properties
Go to Device tab > Select Hardware Ids from Property list
Copy first line (PCI\VEN_10DE&DEV_1BE0&SUBSYS_75071558&REV_A1) and quit

Go to Display.Driver folder which extracted from driver setup.

Open nvami.inf

Find NVIDIA Devices line: [NVIDIA_Devices.NTamd64.10.0...14310]

You will see many lines like below:
%NVIDIA_DEV.0DCE.204C.1043% = Section002, PCI\VEN_10DE&DEV_0DCE&SUBSYS_204C1043

Copy first line until PCI word: %NVIDIA_DEV.0DCE.204C.1043% = Section002,

Paste it to new line above copied line (under NVUDUA_Devices line) and then past hardwareId copied from Device Manager

%NVIDIA_DEV.0DCE.204C.1043% = Section002, PCI\VEN_10DE&DEV_1BE0&SUBSYS_75071558&REV_A1

Do same thing for the line [NVIDIA_Devices.NTamd64.10.0]

You will paste 2 newlines in total to nvami.inf file.

Save inf file to original location and run setup again.

Update

As of 23rd July something changed on Windows 10 and Nvidia setup. Check updates below.
1. If you can't reach Advanced Startup menu then lock your computer. While pressing shift key press power icon on screen and select Restart.

2. Changing inf file may not work on latest driver package. You may get following error:
Package install status code: Exception {0x800f024b - The hash for the file is not present in the specified catalog file. The file is likely corrupt or the victim of tampering.}.
3. Update Windows 10 to the latest version.
Check your version by typing winver to Windows search box to see version. Your version may be older than driver supports.

Install driver from inf file.

Press "Let me pick from a list..." option.
 Find your device in list.

Press "Have Disk" button and find inf file.

Feb 5, 2019

How to use JSON data in Grafana with C# Webapi Backend

1. Install Grafana

1.1 Run Grafana in Docker command line

>docker run -d -p 3000:3000 --name=grafana -e "GF_SERVER_ROOT_URL=http://localhost" -e "GF_SECURITY_ADMIN_PASSWORD=secret"  grafana/grafana

1.2 Install JSON plugin

You can use user interface to install plugins. I will use docker command line interface to install JSON plugin.
Install
>docker exec -it 00798572de34 grafana-cli plugins install simpod-json-datasource
Restart container
>docker restart 00798572de34
Note that 00798572de34 is the container id of the grafana instance in my installation.
>docker ps
CONTAINER ID        IMAGE           ...
00798572de34        grafana/grafana ...

1.3 Login to dashboard

Check Grafana user interface.
http://localhost:3000

username: admin
password: secret

2. Implement Backend

We will prepare some functions for our JSON datasource.
/            should return 200 ok. Used for "Test connection" on the datasource config page.
/search      used by the find metric options on the query tab in panels.
/query       should return metrics based on input.
/annotations should return annotations.

2.1 Webapi implementations

Except the root "/" all functions must have POST method.
 

[HttpPost]
public HttpResponseMessage Query()
{
 return new HttpResponseMessage()
 {
  Content = new StringContent("[{\"columns\":[{\"text\":\"Time\",\"type\":\"time\"},{\"text\":\"Country\",\"type\":\"string\"},{\"text\":\"Number\",\"type\":\"number\"}],\"rows\":[[1234567,\"SE\",182],[1234567,\"DE\",282],[1234567,\"US\",382]],\"type\":\"table\"}]", Encoding.UTF8, "application/json")
 };
}

[HttpPost]
public HttpResponseMessage Search()
{
 return new HttpResponseMessage()
 {
  Content = new StringContent("[ { \"text\" :\"upper_25\", \"value\": 1}, { \"text\" :\"upper_75\", \"value\": 2} ]", Encoding.UTF8, "application/json")
 };            
}

[HttpPost]
public HttpResponseMessage Annotation()
{
 string body = Request.Content.ReadAsStringAsync().Result;
 string headers = Newtonsoft.Json.JsonConvert.SerializeObject(Request.Headers);

 FileHelper fh = new FileHelper();
 fh.FileName = "search.txt";

 fh.AppendFile(body);
 fh.AppendFile(headers);
 return new HttpResponseMessage()
 {
  Content = new StringContent("[{\"text\":\"text shown in body\",\"title\":\"Annotation Title\",\"isRegion\":true,\"time\":\"timestamp\",\"timeEnd\":\"timestamp\",\"tags\":[\"tag1\"]}]", Encoding.UTF8, "application/json")
 };            
}

3. Dashboard & Datasource Setup

3.1 Add dashboard

Go to Grafana web interface and create a new dashboard. Grafana website has a good document for this

3.2 Add datasource

For datasource select newly installed plugin JSON type from the list. We will select most basic settings. Enter a name and datasource url of your API.

HTTP Access
Since we used Docker here, the container must be configured for access to remote site.
If your Grafana server has an access to webapi URL select "Server". Otherwise you can select "Browser".

3.3 Add panel to dashboard

Go to Dashboards > Manage > Select your dashboard.
On right up side there is a button to add a new panel.
Click panel title to see dropdown menu and select Edit.

3.4 Datasource settings

Select your datasource name from Data Source dropdown.
Push "Add Query" button.
If you open Query Inspector panel you will see Grafana tries to reach "/search" url of your webapi.
In metrics textbox enter a column name in your json. In our example I entered @Country.
In Query Inspector panel you will see Grafana tries to reach "/query" URL of your webapi.
If everything is OK you will see your data in a table on the same page.
Save your datasource and return back to your dashboard to see table.

Notes

Response type must be in application/json.
Don't forget to implement root "/" url function for GET method. Grafana checks root function for verifying URL.
Don't forget to add CORS headers if your backend is installed on remote site.

Jan 22, 2019

Decryption function (dotnet & nodejs)

In "Encryption function (dotnet & nodejs)" post I described encryption functions.
In this post you can see decrypt functions.

Nodejs uses "crypto-js" package.
Dotnet uses "System.Security.Cryptography" namespace.

Dotnet

 

public static string Decrypt(string encryptedBase64)
{
 byte[] rawSecretKey = { 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01 };  
 string PassPhrase = "passPhrase";

 MD5CryptoServiceProvider x = new System.Security.Cryptography.MD5CryptoServiceProvider();
 byte[] data = Encoding.ASCII.GetBytes(PassPhrase);
 data.Dump("data");
 
 byte[] passwordKey = x.ComputeHash(data);
 
 RijndaelManaged rijndael = new RijndaelManaged();
 ICryptoTransform rijndaelDecryptor = rijndael.CreateDecryptor(passwordKey, rawSecretKey);
 try
 {
  byte[] encryptedData = Convert.FromBase64String(encryptedBase64);
  byte[] newClearData = rijndaelDecryptor.TransformFinalBlock(encryptedData, 0, encryptedData.Length);
  return Encoding.ASCII.GetString(newClearData);
 }
 catch (Exception ex)
 {
  return null;
 }
}

Node.js

 

 var bytes = CryptoJS.AES.decrypt(encrypted, myRijndaelKey, {
        iv: ivCodeWords,
        mode: CryptoJS.mode.CBC,
        padding: CryptoJS.pad.Pkcs7
    });
 var plaintext = bytes.toString(CryptoJS.enc.Utf8);
 console.log("decrypted text: " + plaintext);