I am speaking on IoTCoast2Coast on ‘Optimize cloud costing by using Azure well-Architected framework’ along with Nik on Wed, October 20, 2021 6:00 PM – 7:30 PM CST
#azure #mvpbuzz #costoptimization Saskatoon Tech Talks
Register below :
If data sources are installed at a long distance from the stakeholders who control the operation, timely notifications are required.
Agriculture, mining, power generation, oil and gas, and other industries can all benefit
Technologies used: Azure Databricks, Python, pyspark, Scala, Power BI
Data sources: Sensors – like heat, humidity, water leakage, electric meters, drones sending data to iot hub.
Solution description: Data from IOT sources are loaded into ADLS Gen2. A checkpoint is created, and autoloader is configured to load and maintain the data. Lastly Azure data bricks bronze, silver & gold tables are created which supplies data to power BI dashboards based on business rules.
Architecture:
Solution Details:
Step 1: Initial configuration, autoloader & load data
Data from various sources are periodically copied to iot-hub, Now, once file is loaded, autoloader tool ingests this data to azure data lake storage gen2 (ADLS Gen2) databricks mount.
Step 2: Create Delta Lake Bronze table
All raw data in csv format can be ingested to bronze table. Basically, it is complete load of all data received.
Step 3: Data processing in Silver Table
This is one of the major steps and in this solution, data processing is performed and cleaned data is saved in silver table.
Step 4: Data processing in Gold Table
In this step, as per business requirements, business logic is applied, and alerts columns are created. This is applied on current load.
Step 5: Dashboard (Power BI):
Gold table is source for Power BI dashboard. This dashboard can help user to sort the existing alert get more details about it. User can also see different analytics on the dashboard.
Based on business mandate, email notification is sent to stake holders
Code Details:
Initial Connection
import json
import csv
import pyspark
from pyspark.sql import SparkSession
configs = {“fs.azure.account.auth.type”: “OAuth”,
“fs.azure.account.oauth.provider.type”: “org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider”,
“fs.azure.account.oauth2.client.id”: dbutils.secrets.get(scope = “ Scope “, key = “<your ID >”),
“fs.azure.account.oauth2.client.secret”: dbutils.secrets.get(scope = “Scope“, key = “Your Key“),
“fs.azure.account.oauth2.client.endpoint”: “https://login.microsoftonline.com/—2222-497b-xxxx-XXXXXX/oauth2/token” }
adlsPath = “<<my-name>@<my-organization.com>>” # Source
mountPoint = “<Mount point Path>” # Upload Path – # Destination
print(f”adls Path: {adlsPath} “)
print(f”Client Secret : {configs} “)
display(dbutils.fs.ls(“Mount point Path “))
Autoloader configuration
# Run the following code for autoloader configuration
checkpoint_path = ‘/Mount point Path/_checkpointname/’
import json
import csv
from pyspark.sql.functions import input_file_name
cloudfile = {
"cloudFiles.format": "csv",
"cloudFiles.schemaEvolutionMode": "addNewColumns",
"cloudFiles.inferColumnTypes": "true",
"cloudFiles.includeExistingFiles": "true",
"cloudFiles.allowOverwrites": "false",
"cloudFiles.schemaLocation": "/Mount point Path /_checkpointname/",
"rescueDataColumn":"_rescued_data",
"cloudFiles.useNotifications":"false"
}
# Set up the stream to begin reading incoming files from the Mount point Path location.
df = (spark.readStream.format("cloudFiles").options(**cloudfile).load('/Mount point Path/StreamingLog) withColumn("filePath",input_file_name())) # this adds a column with file name
Create Dataframe using new uploaded data
Here checkpoint is tracking or keeping a record of all the files that are uploaded to mountpoint. newly uploaded files those don’t have any record in checkpoint is loaded at DELTALAKE_BRONZE_PATH
# Start the stream & write the data
DELTALAKE_BRONZE_PATH = "dbfs:/FileStore/Bronze_StreamingLog "
df.writeStream\
.format("delta")\
.outputMode("append")\
.option("checkpointLocation", "/Mount point Path /_checkpointname/")\
.trigger(once=True)\
.start(DELTALAKE_BRONZE_PATH)
Create Delta Lake Bronze table
# Register the SQL table in the database
spark.sql(f"CREATE TABLE IF NOT EXISTS <Bronze_Tablename> USING delta LOCATION '{DELTALAKE_BRONZE_PATH}'")
# Read the table
streaminglog_stats = spark.read.format("delta").load(DELTALAKE_BRONZE_PATH)
display(streaminglog_stats)
Silver & Gold Table: Once data is uploaded into the bronze table, all data cleaning and ETL can be performed on it and clean data can be saved into the silver table.
# Configure destination path
DELTALAKE_SILVER_PATH = "dbfs:/FileStore/Silver_StreamingLog "
# Write out the table
streaminglog_stats.write.format('delta').mode('overwrite').save(DELTALAKE_SILVER_PATH)
# Register the SQL table in the database
spark.sql("CREATE TABLE if not exists <Silver_Tablename> USING DELTA LOCATION '" + DELTALAKE_SILVER_PATH + "'")
# Read the table
streaminglog _stats = spark.read.format("delta").load(DELTALAKE_SILVER_PATH)
display(streaminglog _stats)
After formation of silver table, all business rules are applied, and gold table is created. This is the source for all reporting, dash boards and reporting tools, (Power BI) in our case.
It’s a privilege to review the technical book “Application Delivery and Load Balancing in Microsoft Azure: Practical Solutions with NGINX and Microsoft Azure.”
My role was to assist the authors in identifying and correcting any major technical or structural errors, as well as providing up-to-date technical information.
So happy to contribute as volunteer on ‘Global Power Platform Bootcamp 2021’ with my friends!!
#GlobalPowerPlatformBootcamp#GPPB2021#PowerPlatform
Thanks everyone..
I got wonderful opportunity to be a guest on January 13 at MVP LIVE show.
Please refer below link:
https://www.c-sharpcorner.com/events/discussion-with-a-canadian-mvp-mvp-show-ft-deepak-kaushik
Thanks to 3K audience who joined me on LIVE show!!
I am so glad to be featured as one of the top 10 speaker of ‘Year 2020’ on C# Corner
https://www.c-sharpcorner.com/article/top-10-c-sharp-corner-speakers-of-year-2020/
C# Corner organized 197 events under 46 chapters from around the world. These events include in-person events, webinars, and live shows
Thank you, chapter leaders, speakers, and members, for the success of these events.
Kusto is a service for storing and running interactive analytics over Big Data based on Microsoft Azure infrastructure.
Thanks everyone to attend the Webinar on Dec 12th, 2020.
Here is recording:
AGENDA
Getting Started with Kusto
Kusto Explorer
Entity Types
Query Statements
Join here:
https://teams.microsoft.com/l/meetup-join/19%3a1df1264620c04b6084f000557
SPEAKERS
Deepak Kaushik AND Shahriar Nikkhah
DEEPAK KAUSHIK @THINKFORDEEPAK
Deepak is a Microsoft Azure MVP and C# corner MVP. He is currently working on architecting and building solutions around Microsoft Azure. He is passionate about technology and comes from a development background. He has also led various projects in the Infrastructure as a Service (IaaS) and Platform as a Service (PaaS). Deepak recently worked with Public Sector client to migrate ‘On-Prem’ DWH to Azure SQL Data warehouse and reports migration from SSRS reports to POWER BI. His breadth and depth of knowledge have enabled him to lead the development of various products/solutions around Microsoft Azure. Deepak is a knowledgeable and sought-after speaker within IT circles and consulted regularly by companies formulating their Cloud strategies.Deepak founded Regina/Saskatchewan ‘C# Corner Chapter/ User Group’ in 2016 in the Regina/ Saskatchewan area where we organizes regular meetups and webinars!!You could find his Sessions/ Recordings at https://channel9.msdn.com – Channel9, C# Corner, Blog and Deepak Kaushik -Microsoft MVP YouTube Channel.
SHAHRIAR NIKKHAH C# CORNER MVP
Snr Technical Team Lead/Senior BI Consultant, Big Data, IoT Hub, Azure Stream Analytics, Power BI, SSIS, SQL Tabular DM, MCSA: BI Reporting, MCSE: Data Management and Analytics SQL2016, MCSA: SQL 2016 Database Development, MCP, Microsoft Certified Professional (SQL2012), MCPS: Microsoft Certified Professional, MCITP, Database Developer 2008, MCTS SQL 2008 Database Development, MCTS SQL 2008 Implementation and Maintenance, MCTS SQL 2005********Career Objective To obtain a senior level Business Intelligence Designer/Developer/Lead position (SQLServer, Big data, Stream Analytics, IoT Hub, SSIS, Power BI, SSAS, SSRS, Datazen, Tableau, Power Pivot, Power View, etc…).********Experience summary Focused on Big data, Data Warehouse, BI DW design, Business Intelligence Remote team lead and work experience, in Canada, US and EU for BI and ETL design teams. Senior level experience in Transactional DW and BI DW design projects. Coaching, mentoring, Leading, providing BI presentation and BI solutions for, end users, IT teams and management teams********Technical skills Database: SQL Server 2017 down to 7.0, Oracle 9.x, Teradata, Azure Tabular Model, SSAS 2008-2012-2014, SQL Tabular (BISM), Access. BI Tools: SSAS, SSIS, Power Query, PowerPivot, Power View, Power Map, Power BI, Analytic Tools, SSRS, SharePoint, Excel Pivot tables, Datazen Publisher, Tableau. Languages: T-SQL, M Language, DAX, MDX, PowerShell, VB.Net, C#. Other: VS, Cloud Base Technologies: SQL Azure, Power BI Desktop/Design, Power BI Portal/Desktop, O365, Azure Data Factory (ADF), Datazen Publisher, Azure Resource Group, Azure Data Gateway, Data Management Gateway, oAuth, Azure PowerShell.********Certification, Awards, Publication, Contribution MCSA: BI Reporting — Certified 2018 MCSE: Data Management and Analytics — Certified 2018 MCSA: SQL 2016 Database Development – Certified 2018 Power BI forum Contributor at http://www.community.powerbi.com (Click on me for my id, “SNik”) MCP (MS-Certified Professional) SQL 2012, (See attached Microsoft CP Transcript) Co-authors of “SQL MVP Deep Dive 2” 2011 (Support the children of Operation Smile) Microsoft SQL MVP 2010 Award (Microsoft Most Valuable Professional) MCITP (MS-Certified IT Professional), MCTS (MS-Certified Technology Specialist) 4 Star rating at “SQL Server Integration Service (SSIS)” forum online at msdn.Microsoft.com
I’m so Thankful, honored, and excited to receive/ renewed C# Corner Most Valuable Professional (MVP) Award for the 2020-2021 for 4th time!!
I am sincerely humbled to be part of such an incredible CSharpCorner community!
This year I have talked at 11 conferences & User Groups where approx. 4000+ people attended, wrote many blogs & articles at my website and different technical portals!
In addendum, I talked at 10+ webcasts / podcasts / Live Interviews and reached to 3000+ attendees, Made a lot of #Azure Videos, mentored couple of students/ professionals on #Azure / suggestion / advice / resources and #saskatchewan Job Market insights!
Again, Many Thanks CSharpCorner Team!
Wohoo 🙌 !!
I’m so Thankful, honored, and excited to receive/ renewed Microsoft Most Valuable Professional (MVP) Award for the 2020-2021 in #AZURE Category for 3rd time!!
I am sincerely humbled to be part of such an incredible community, celebrating #CanadaDay while being recognized for my contributions!!
This year I have talked at 11 🌎 conferences & User Groups where approx. 4000+ people attended, wrote many blogs & articles at my website and different technical portals!
In addendum, I talked at 10+ webcasts / podcasts / Live Interviews and reached to 3000+ attendees, Made a lot of #Azure Videos, mentored couple of students/ professionals on #Azure / suggestion / advice / resources and #saskatchewan Job Market insights!!
Again Many Thanks 🎉👏
#MVPBuzz#MVPAward#CdnMVP#MVP#MicrosoftMVP
This is my 3rd MVP Award and I am very grateful and appreciative for this honor and for the various opportunities provided to me over time.
Thank you very much to each and every one of you for making me successful in my efforts as a MVP, IT Professional, and community contributor, and for providing me with the valuable resources and networking opportunities. Thank you!
Deepak Kaushik -> Cloud Blooded Beast 