deep dive openshitt on azure & .net core on openshift
TRANSCRIPT
Deep DiveOpenShift on Azure
&.NET Core on OpenShift
Takayoshi Tanaka @TanakaTakayoshi
Red Hat K.K. (Japan) [email protected]
Notes:This slide is available online.
As I have tested at OCP 3.5 and .NET Core 2.0 preview2,something will be changed at the latest OCP 3.6 and .NET Core 2.0 RTM.
If you have any question or comments, feel free to contact me:
email: [email protected]
Twitter: @TanakaTakayoshi
BackgroundRed Hat K.K. (Japan)◦ Software Maintenance Engineer
◦ OpenShift◦ Red Hat solutions on Azure◦ .NET Core on RHEL
Personal◦ Microsoft MVP for VSDT
◦ C# Lang, .NET Core on Linux
◦ Blogs:◦ Red Hat Developers◦ Personal Blog “Silver light and Blue sky”
VSDT: Visual Studio & Development Technologies
Goal◦ Learn about OpenShift on Azure Reference Architecture
◦ How to integrate Azure Features with OpenShift
◦ .NET Core 2.0/integrating OpenShift features with ASP.NET Core
Reference ArchitectureDocument is now available◦ Deploying Red Hat OpenShift Container Platform 3 on Microsoft Azure
OpenShift Ansible - Azure ARM Template
Installation Summary
1• ARM Template for Azure Resources (VM, LB, NW…)
2
• Custom Script Extension with ARM
• generate config. files & execute ansible
3• Ansible Installer for OpenShift
Known Issue
Available only in the Azure Marketplace VM
• duplicated billing. Custom image (.vhd) is on the roadmap.
No official Red Hat is available (self-support only)
• You should troubleshoot by yourself.
The OpenShift VM configuration is fixed
• 3 masters with etcd (same hosts), 3 infra nodes, 3+ nodes, 1 bastion
構成図
構成図
構成図
構成図
構成図
VMs1 bastion
3 masters with etcd
3 infra nodes (router/docker registry)
3+ nodes
Support request required for increasing cpu core limit.
This limitation is due to design of ARM template.
You can install all-in-one OpenShift on 1 host (not supported)
Examples: Integrating Azure FeaturesAvailability Set
Azure Load Balancer◦ master endpoint
◦ backend is a group of masters
◦ routing endpoint◦ backend is a group of infra nodes (routers)
Azure VHD for Persistent Volume (PV)◦ Virtual Hard Disk for Azure VM (VHD)
◦ Dynamic provisioning Available at OCP 3.5+
How does Azure VHD for PV work?
node service
/etc/azure.conf
1. node service receivesVolume Mount request
2. Load azure.conf(API auth etc)
3. (if dynamic provisioning)Create an empty VHD
4. Mount VHD to Azure VM5. Create filesystem if needed6. Mount filesystem to container
Depending on kubernetess Azure Volume Plugin
VHD
attach
How to configure azure.conf
See the document for more detail.
Easy 3 steps with Azure CLI 2.0
$ az account list -o json//Retrieve tenantID & id
$ az group show --name <ResourceGroupName> -o json//Retrieve id & location
$ az ad sp create-for-rbac --name <ResourceGroupName> --role contributor \--scopes "<Resource Id>“ -o json//Retrieve appId, password
Azure VHD for PV NotesManaged Disk is unavailable◦ kubernetes Azure Disk plugin is not supported Managed Disk
Be sure to confirm VM name to hostname◦ Also specification of kubernetes plugin
Configure DNS yourself◦ VMs can be communicate with their VM name.
◦ If not using Azure internal DNS
◦ If using VNET peering or other
More Azure FeaturesAzure Active Directory Open ID Connect◦ authentication for master◦ LDAP integration with AAD+AAD DS or AD is also available.
Azure Blob Storage for OpenShift internal docker registry◦ object storage is suitable for docker registry storage
Azure File Storage◦ File storage is also available for PV◦ Linux kernel CIFS module with SMB 3 is still experimental
Operation Management Suite integration◦ Log Analysis also available for containers
How to set up OpenID connect?Create Azure AD App using the Microsoft Azure portal
How to set up LDAP auth with AD?Option A) AAD + AAD DS + (VNET peering or VNET-to-VNET VPN)
* AAD DS only supports Classic VNET and requires private network from ARM VNET.
AAD AAD DS
classic VNET ARM VNET
OCP
master
ldap://xx.xx.xx.xx/
AAD DS configuration example- name: "aad_ds_provider"
challenge: true login: true mappingMethod: claim provider:
apiVersion: v1kind: LDAPPasswordIdentityProviderattributes:
id: - dnemail: - userPrincipalNamename: - cnpreferredUsername: - cn
bindDN: "cn=adadmin,ou=AADDC Users,DC=example,DC=onmicrosoft,DC=com" bindPassword: "<password>" insecure: trueurl: "ldap://XXX.XX.XX.XX/OU=AADDC Users,DC=example,DC=onmicrosoft,DC=com?
userPrincipalName?sub?(memberof=CN=ocpgroup,OU=AADDC Users,DC=example,DC=onmicrosoft,DC=com)"
master-config.yaml
ou: AADDC UsersAAD default OU
userPrincipalName will be email
How to set up LDAP auth with AD?Option B) on premise AD + VPN
Connect on premise Network and ARM Network with VPN.
AD
on premise NW ARM NW
OCP
master
ldap://xx.xx.xx.xx/
Storage Technology ComparisonType References Notes
Azure Blob Storage
Object storage
Extended Registry ConfigurationMicrosoft Azure storage driverDeploying Your Own Private Docker Registry on AzureAzure Blob Storage
Only Available for docker registry storage
Azure VHD Filesystemon external Disk
Persistent Storage Using Azure DiskConfiguring for AzureAbout disks and VHDs for Azure Linux VMs
Depends on k8s plugin
Azure File Storage
NFS Persistent Storage Using Azure FileConfiguring for AzureHow to use Azure File Storage with Linux
Depends on k8s pluginExperimental.
External NFS service
NFS N/A Should maintain yourself or buy 3rd
party service.
Operation Management Suite (OMS)Log Analysis & other features for on-premised to cloud
Containers (Preview) solution in Log Analytics now support OpenShift
Installing OMS AgentAdding OMS agent directly on Linux Host
Or, install agent as a OpenShift daemonset
Container solution
Future: Windows Container?No roadmap: Windows Container
kubernets has roadmap for working with Windows Container
“Capability” is existing.
.NET Application Model
35
.NET Core Inside
36
.NET Core App (C#/VB)
IL Assembly (exe, dll)
Roslyn CoreFX(.NET Core
Class Library)
ManagedUnmanaged
OS
Native ABI etc
(F#)compile
r
CoreCLR(.NET Core Runtime)
.NET Core SDK T
oo
ls
.NET Core on OpenShifts2i build◦ “Source code in the Git repo” To “docker Image”
◦ can run out of OpenShift
s2i image◦ parameters for simple customize
◦ more customization are available with s2i scripts
Template project◦ Start .NET Core on OpenShift with few clicks at the portal
◦ All in one: deploymentconfig, service, route etc…
s2i build & deploy flow
builder pod
SCM(git) internal registry
deployer pod
pod
deploymentConfigbuildConfig
$ dotnet build$ dotnet publish
$ dotnet <dll>
.NET Core 2.0 launch start today!rh-dotnet supports csproj at .NET Core 2.0◦ rpm version will be available
◦ s2i for .NET Core 2.0 & ASP.NET Core 2.0
◦ Runtime image & s2i image (s2i image only at 1.x)
More new features coming◦ Announcing .NET Standard 2.0◦ Announcing .NET Core 2.0
◦ Introducing ASP.NET Core 2.0
◦ Announcing Entity Framework Core 2.0
Use Case Examples◦Schedule Jobs with .NET Core◦Switching Configuration for Dev & Prod Environment◦Razor Page & C# 7.1◦Redis for HTTP Session storage with multi pods
Notes:All examples are built on .NET Core 2.0 preview.We’re actively working on it now.
Schedule Job with .NET Core
Run .NET Core Console App as a cron job: Cron JobsExample Repository
Web portal does not support cron jobs, so use the CLI.
$ oc create imagestream cronjobexample$ oc create -f cronjob-buildconfig.yaml$ oc create -f cronjob.yaml
Schedule Jobs with .NET Corespec:
containers:- command:
- /opt/rh/rh-dotnet20/root/bin/dotnet- bin/Release/netcoreapp2.0/CronJobExample.dllimage: 172.30.142.2:5000/london-openshift/cronjobexample:latestimagePullPolicy: Alwaysname: lipsum-cronresources: {}
restartPolicy: NeversecurityContext: {}terminationGracePeriodSeconds: 30
schedule: '*/1 * * * *'
command to execute:should be the full path
command to execute:*scl should be enabled.
To be fixed in my example
image should be specified with full URLOCP 3.6 will support imagestreamtag.Replace 172.30.142.2:5000 with your
internal registry’s IP and port
Switching Configuration for Dev & Prod Environment
How to treat different environments with one code◦ Connect to different database
◦ Use Redis as a cache - only in a production environment
◦ Integrate with a different OpenID account
Use Environment feature in ASP.NET Core◦ Specified by environment variables.
Configuration can be injected specific to each environment.
Switching with Environment
Startup class• constructor• Configure method• ConfigurreService method
Check IHostingEnvironment
Can’t inject IHostingEnvironment into ConfigerService method
Switching ConfigureXXXServicesConfigureXXXServices
Switching ConfigureXXXConfigureXXX
Inject configuration from Environment Variable
Loading configuration from Secretuse OpenShift secret feature.
Razor Page + C# 7.1RazorPage:◦ Simpler application than original MVC: “Page-focused scenarios”
◦ WebMatrix like easy development◦ Razor Page is enabled with MVC
C# 7.1:◦ available at .NET Core 2.0 & ASP.NET Core 2.0
◦ C# 7.1 in Razor page is not working at Preview 2 by bug (see issue)◦ It should be fixed at 2.0 RTM.
HTTP session for multi pods By default:◦ Sticky session: request goes to the same pod in same user session ◦ HTTP session is stored in the memory of each pod◦ HTTP session is encrypted by pod specific key
It means:
When a pod has died, a user session will be lost.
How to keep HTTP session
IDistributedCache & IDataProtectionIDistributedCache◦ Provide distribution cache
◦ Available for storing session
◦ ASP.NET Core team provides SQLServer and Redis
IDataProtection◦ Provide key management for encryption
◦ Encrypt http session
◦ By default, generate machine (=pod) specific key and store in local file
◦ ASP.NET Core team provides NFS, Redis and AzureStorage (Preview)
machine A
pod B
pod C
ASP.NET Core
ASP.NET Core
ASP.NET Core
Session Data A’
Each pod has a different key.Can’t decrypt session data When loading another pod from a different sessionload with same id.
~/.aspnet
default implementation of IDataProtection
machine A
machine B
machine C
ASP.NET Core
ASP.NET Core
ASP.NET Core
Session Data
Use DataProtection.Redis
Configuration for Redispublic void ConfigureServices(IServiceCollection services){
// You can retrieve this connection string from Azure Portal.var conn = Configuration["REDIS_CONNECTION_STRING"];
var redis = ConnectionMultiplexer.Connect(conn);
services.AddDataProtection().PersistKeysToRedis(redis, "DataProtection-Keys");
services.AddDistributedRedisCache(option =>{
option.Configuration = conn;option.InstanceName = "master";
});services.AddSession();
}
httpsession.redis.cache.windows.net:6380,password=<password>,ssl=True,abortConnect=False
High Level Debugging .NET Core
58
GDB/LLDB
MICore
MIDE/Engine.Impl
MIDE(*)/AD7.Impl
VSCode Debugger
AD7 Interface
*MIDE: MIDebugEngine: GitHub repository*vsdbg can be used only in VS products and might not be distributed.
MIText
VS Debugger
VS Debugger Engine vsdbg
(closed license*)
windbg
See more:Architecture of MIEngine
Remote Debugging .NET Corevsdbg provided by Microsoft◦ only trusted communication is required
◦ SSH is generally available
◦ VS remote debugger tools is also available on Windows
◦ Due to the license limitation, VS products (VS, VS Code, VS for mac) are only available for debugging.
* Low level debugger is provided by Red Hat◦ sos
◦ Not providing graphical debugger interface
59
Remote debugging toa container on OpenShift“oc rsh” is available instead of ssh
vsdbg should be manually installed◦ install script is unavailable as s2i image doesn’t have unzip
◦ download vsdbg on local and rsync
see more detail in my wiki
Remote debug from Visual Studio Code.vscode/launch.json
61
{"name": ".NET Core Docker Remote Attach","type": "coreclr","request": "attach",“processId”: “1”, "pipeTransport": {
"pipeProgram": “oc",“pipeArgs”: [ “rsh”, “-T”, “[email protected]”], “quoteArgs”:false, “debuggerPath”: “/opt/app-root/src/vsdbg/vsdbg”, "pipeCwd": "${workspaceRoot}"
},"sourceFileMap": {
"/opt/app-root/src": "${workspaceRoot}“}
}
SummaryOpenShift on Azure◦ Reference Architecture is a good place to start.
◦ More Azure features available-- Authenticating with OpenID and others
.NET Core 2.0/ASP.NET Core 2.0 on OpenShift◦ csproj support
◦ cronjob for .NET Core console app
◦ OpenShift secret & configuration. ASP.NET Core environment
◦ Remote debugging