2.2.4. LAB 4 – DELETE AN SSLO CONFIGURATION¶

One of the benefits of the new SSLO architecture is that configurations can be edited, deployed and re-deployed without affecting existing traffic flows. For this capability, the SSLO packaging is now broken into separate independent components. When deleting a defined topology, most of the attached components are also deleted. However, some objects, particularly those that can be consumed by multiple topologies, are not automatically deleted. This lab explores the different methods for deleting SSL Orchestratorobjects.

2.2.4.1. Step 1: Deleting a topology¶

Deleting a topology will also delete any relying Interception Rules. The deletion process performs a complex set of REST-based tasks, therefore only one topology can be deleted at a time. In the SSLO UI, select a topology and click the Delete button. Confirm that both the topology and respective interception rules are removed.

2.2.4.2. Step 2: Deleting other objects¶

While deleting a topology also removes its respective interception rules, it does not remove the other objects - services, service chains, security policies and SSL settings. These can all be removed individually, however must be deleted in a hierarchical order. Once the topology and interception rules have been deleted,

SSL Settings can be deleted any time
Delete any unused Security Policies
Delete any unused Service Chains
Delete any unused Services

2.2.4.3. Step 3: Deleting everything¶

To completely remove the SSLO configuration and start from scratch,

In the SSLO UI, click Delete Configurations and then click OK. This process will take some time as SSLO walks through all of the objects and dependencies to remove all configurations.
Under the iApps menu, Application Services, Applications LX – un-deploy any remaining SSL orchestrator objects. If using any other Guided Configuration engine (ex. Access GC), ensure that only SSLO objects are deleted here.
Under the iApps menu, Templates, Templates LX – delete all of the SSL Orchestrator templates.
Under the iApps menu, Package management LX – delete the SSL Orchestrator package.

The next time the SSL Orchestrator configuration menu is accessed, SSLO will automatically restore the on-box package.

2.2.4.4. Optional: Deleting everything…the hard way¶

In the unlikely event that the above steps do not work, and some SSLO objects remain and cannot be deleted, one of the following steps can be used,

If the topology and interception rules are gone but other objects remain and will not uninstall in the SSL Orchestrator UI, in the BIG-IP UI navigate to iApps -> Application Services -> Applications LX. The remaining objects will all be here in states of deployed (green), undeployed (gray), and error (red). Delete any objects in an error state and toggle the other objects from deployed to undeployed and back until they enter an error state and can also be deleted.
If the above fails, the following script can be used to automate destruction of SSLO objects.
- Copy the script to the BIG-IP (ex. cleaner.sh)
- Chmod the script to give it execute privileges: chmod +x cleaner.sh
- Execute the script: ./cleaner.sh
- It will typically be necessary to execute the script several times to get through dependencies. It is completely done when the script returns quickly with no additional output. Validate that all SSLO objects are gone from the BIG-IP UI under the Local Traffic and Network sections.
- Under the iApps menu, Application Services, Applications LX – un-deploy any remaining SSL orchestrator objects. If using any other Guided Configuration engine (ex. Access GC), ensure that only SSLO objects are deleted here.
- Under the iApps menu, Templates, Templates LX – delete all of the SSL Orchestrator templates.
- Under the iApps menu, Package management LX – delete the SSL Orchestrator package.
If the above fails, manually clear the REST database from the command line,
- Break any HA configuration
- Issue the ‘clear-rest-storage [options]’ command, where the options are “-l” (lowercase L) to delete the restjavad log files as well as the stored state, and “-d” to reset the system configuration to default. This command will remove all SSL Orchestrator objects from the restnoded database. After issuing this command, follow with ‘bigstart restart restnoded’ and ‘bigstart restart restjavad’, clear the browser cache, log out and back in.
- Issue the ‘tmsh delete sys application service recursive’ command to also delete any remaining SSL Orchestrator application service objects.
- Once all SSLO objects have been removed, also uninstall the SSLO RPM package under the iApps menu, Package management LX – delete the SSL Orchestrator package.
- Rebuild HA and redeploy SSLO by navigating to the SSL Orchestrator configuration UI. On first visit it will automatically restore the on-box package.

2.2.5. TROUBLESHOOT SSLO¶

While the SSL Orchestrator product has certainly evolved, as with anything in the computing world, problems are usually inevitable and poorly timed. In the event that an SSL Orchestrator configuration has failed, or that it has succeeded but not behaving as expected, the following troubleshooting tools should be useful.

2.2.5.1. Step 1: Test the configuration¶

It is important to first define “normal” behavior. If the SSL Orchestrator deployment process was successful, it will be possible to access remote Internet sites from the client workstation without issue, and HTTPS sites appear to have a locally-trusted, re-issued server certificate. This would be considered normal behavior. If any of these do not happen, use the tools below to troubleshoot.

2.2.5.2. Step 2: Troubleshoot¶

Below is a reasonably-ordered list of troubleshooting steps.

If the SSL Orchestrator deployment process fails, review the ensuing error message. It would be impossible to list here all of the possible error messages and their meanings, but often enough the messages will reveal the issue.
Re-review the lab steps for any missing or misconfigured settings.
Enable debug logging in the SSL Orchestrator configuration. Tail the APM log from a BIG-IP command line or from the logs page in the management UI. Debug logging will very often reveal important issues. Specifically, it will indicate traffic classification matches, mismatches or deployment issues.

tail -f /var/log/apm

tail -f /var/log/restnoded/restnoded.log

tail -f /var/log/restjavad.0.log
If the SSL Orchestrator deployment process succeeds, but traffic isn’t flowing through the environment made evident by lack of access to remote sites from the client:
- Ensure that the client is properly configured to either default route to the ingress VLAN and self- IP of the BIG-IP for transparent proxy access or has the correct browser proxy settings defined for explicit proxy access.
- Ensure that traffic is flowing to the BIG-IP from the client with a tcpdump capture at the ingress interface.
- Review the LTM configuration created by the SSL Orchestrator. Specifically, look at the pools and respective monitors for any failures.
- Isolate service chain services. If at least one service chain has been created, and debug logging indicates that traffic is matching this chain, remove all but one service from that chain and test. Add one service back at a time until traffic flow stops. If a single added service breaks traffic flow, this service will typically be the culprit.
- If a broken service is identified, insert probes to verify inbound and outbound traffic flow. Inline services will have a source (S) VLAN and destination (D) VLAN, and ICAP and receive only services will each have a single source VLAN. Insert a tcpdump capture at each VLAN in order to determine if traffic is getting to the device, and if traffic is leaving the device through its outbound interface.
- If no service chains are defined, it may be necessary to remove all of the defined services and re- create them one-by-one to validate flow through the built-in All chain. If a broken service is identified, insert tcpdump probes as described above.
- If traffic is flowing through all of the security devices, insert a tcpdump probe at the egress point to verify traffic is leaving the BIG-IP to the gateway router.
  
  tcpdump -I 0.0:nnn -nn -Xs0 -vv -w <file.pcap> <any additional filters>
- If traffic is flowing to the gateway router, perform a more extensive packet analysis to determine if SSL if failing between the BIG-IP egress point and the remote server.
  
  Then either export this capture to WireShark are send to ssldump:
  
  ssldump -nr <file.pcap> -H -S crypto > text-file.txt
- If the WireShark or ssldump analysis verifies an SSL issue:
  - Plug the site’s address into the SSLLabs.com server test site at: https://wwww.ssllabs.com/ssltest
  This report will indicate any specific SSL requirements that this site has.
  - Verify that the SSL Orchestrator server SSL profiles (two of them) have the correct cipher string to match the requirements of this site. To do that, perform the following command at the BIG-IP command line:
    
    tmm –clientciphers ‘cipher string as displayed in server ssl profiles’
  - Further SSL/TLS issues are beyond the depth of this lab guide. Seek assistance.
  - If all else fails, seek assistance.

2.2.6. APPENDIX – COMMON TESTING COMMANDS¶

The following are some simple, but powerful commands that are useful in developing and troubleshooting SSL visibility projects.

2.2.6.1. Control the SSLFWD certificate cache¶

The behavior of the SSL Forward Proxy changes after a certificate is cached, which will make it difficult to troubleshoot some issues. The following allows you to both list and delete the certificates in the cache.

tmsh show ltm clientssl-proxy cached-certs clientssl-profile [CLIENTSSLPROFILE] virtual INGRESSTCPVIP] tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [CLIENTSSLPROFILE] virtual INGRESSTCPVIP]

2.2.6.2. Isolate SSLO traffic¶

Any given website will be full of images, scripts, style sheets, and very often references to document objects on other sites (like a CDN). This can make troubleshooting very complex. The following cURL commands allow you to isolate traffic to a single request and response.

curl -vk https://www.bing.com

curl -vk –proxy 10.30.0.150:3128 https://www.bing.com

curl -vk –proxy 10.30.0.150:3128 - -location https://www.bing.com

Optionally, between each cURL test, delete the certificate cache and start logging:

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [CLIENTSSLPROFILE] virtual INGRESSTCPVIP] && tail -f /var/log/apm

2.2.6.3. Debugging¶

There is simply nothing better than debug logging for troubleshooting SSL intercept issues. The SSL Orchestrator in debug mode pumps out an enormous set of logs, describing every step along a connection’s path. Remember to never leave debug logging enabled.

tail -f /var/log/apm

2.2.6.4. Packet capture¶

Second only to debug logging, packet captures are crucial to troubleshooting any network-dependent issue.

tcpdump -lnni [VLAN] [-Xs0]

In-line services create “source” (S) and “destination” (D) VLANs, and ICAP and receive-only services attach to existing VLANs. Drop a probe at each point in the path and observe flow.

2.2.6.5. SSL inspection¶

ssldump -AdNd -i [VLAN] port 443 <add additional filters>

tcpdump -i 0.0:nnn -nn -Xs0 -vv -w <file.pcap> <and additional filters>

ssldump -nr <file.pcap> -H -S crypto > text-file.txt

TLS is rarely the issue, but a sight or configuration error may render some sites inaccessible.

2.2.6.6. Control the URL Filtering database¶

To show the current status of the database: tmsh list sys url-db download-result

To initiate (force) the URL DB to update: tmsh modify sys url-db download-schedule all stats true download-now true

To verify that the URL DB is actively updating: tcpdup -lnni 0.0 port 80 and host 204.15.67.80

2.2.6.7. External testing¶

Plug the site’s address into SSLLabs.com server test site at https://wwww.ssllabs.com/ssltest to see if the site has any unusual SSL/TLS requirements.

2.2.7. APPENDIX – ROUTING CONSIDERATIONS FOR LAYER 3 DEVICES¶

SSL Orchestrator sends all traffic through an inline layer 3 or HTTP device in the same direction – entering through the inbound interface. It is likely, therefore, that the layer 3 device may not be able to correctly route both outbound (forward proxy) and inbound (reverse proxy) traffic at the same time. Please see the appendix, “Routing considerations for layer 3 devices” for more details. For example, in a simple Linux-type environment there would be two routes needed for SSLO:

The default gateway to send traffic back to SSLO through the service’s outbound interface
A static return route to allow client traffic to return through the service’s inbound interface Example:

In the above, configured for an outbound traffic flow, the default gateway is on the outbound side interface (eth2), with a static route for 10.1.10.0/24 (client-sourced) traffic flowing back through the inbound interface (eth1). An inbound flow, however, would require the opposite:

Destination	Gateway	Genmask	Flags	Metric	iFace
default	198.19.64.7	0.0.0.0	UG	0	eth1
10.1.10.0	198.19.64.245	255.255.255.0	UG	o	eth2

There are generally a few options for handling inbound and outbound traffic flows:

Do not use the same layer 3 device for inbound and outbound flows – the simplest option, but not always possible in some environments.
Create a policy route, if the device supports it, to create multiple gateways. We will explore the second and second options below.

2.2.7.1. Configuring a policy route on the layer 3 device¶

If a service supports it, policy routing allows you to create multiple gateways on a layer 3 (routed) device. In lieu of creating separate inbound and outbound services, and service chains for a single L3 device, all traffic in this use case still flows through the inbound side interface, but the policy route will effectively steer traffic in the correct direction. Policy routing can be a complex topic in and of itself, and each security product will have its own way of configuring policy routing anyway, so it cannot be covered in total in this guide. Please refer to product-specific documentation to learn more about your policy routing options.

The following is an example script to enable a policy route on a generic Linux device (most of which have iproute2 installed by default). In the script, it is only necessary to modify the top eight variables, defining attributes of the inbound and outbound networks. Once complete, chmod the script to make it executable, test it, and then call it from a startup process like /etc/rc.local or /etc/init.d/rc.local. If the script is successful, you should be able to send inbound and outbound SSLO traffic flows through this device.

#!/bin/bash

## Inbound interface
inbound_interface=eth1.10
inbound_ip=198.19.64.65 inbound_mask=25
inbound_gw=198.19.64.7

## Outbound interface
outbound_interface=eth1.20
outbound_ip=198.19.64.130 outbound_mask=25
outbound_gw=198.19.64.245

### ---------------------------------------------- ###

### ---------------------------------------------- ###

## static table names
inbound_table=av_in
outbound_table=av_out

## function to get network from mask and IP get_network () {

IFS=. read -r io1 io2 io3 io4 <<< "$2"

set -- $(( 5 - ($1 / 8) )) 255 255 255 255 $(( (255 << (8 - ($1 %8))) & 255 )) 0 0 0
[ $1 -gt 1 ] && shift $1 || shift
NET\_ADDR="$((${io1} & ${1-0})).$((${io2} & ${2-0})).$((${io3} & ${3-0})).$((${io4} & ${4-0}))"

echo "$NET_ADDR"

}

## stop if iproute2 is not installed
if ! [ -d "/etc/iproute2/" ]; then
   echo "iproute2 policy routing is not available on this system - exiting" exit
fi

## create the ipproute2 tables

if ! grep -q ${inbound_table} /etc/iproute2/rt_tables; then echo "200
   ${inbound_table}" >> /etc/iproute2/rt_tables
fi

if ! grep -q ${outbound_table} /etc/iproute2/rt_tables; then echo "201
   ${outbound_table}" >> /etc/iproute2/rt_tables
fi

## get the inbound and outbound networks from function
inbound_net=$(get_network ${inbound_mask} ${inbound_ip})
outbound_net=$(get_network ${outbound_mask} ${outbound_ip})

## create policy routes
ip rule add iif ${inbound_interface} table ${inbound_table}
ip rule add iif ${outbound_interface} table ${outbound_table}
ip addr add ${inbound_ip}/${inbound_mask} brd + dev ${inbound_interface}
ip addr add ${outbound_ip}/${outbound_mask} brd + dev ${outbound_interface}
ip route add ${inbound_net}/${inbound_mask} dev ${inbound_interface} src ${inbound_ip} table ${inbound_table}
ip route add ${outbound_net}/${outbound_mask} dev ${outbound_interface} src ${outbound_ip} table ${outbound_table}
ip route add ${outbound_net}/${outbound_mask} dev ${outbound_interface} src ${outbound_ip} table ${outbound_table}
ip route add default via ${inbound_gw} table ${outbound_table}

2.2.8. APPENDIX – DEMO SCRIPTS¶

2.2.8.1. Lab 1 demo script¶

Configuration review and prerequisites

Optionally define DNS, NTP and gateway route
Click Next

Topology Properties

Name - some name
Protocol: Any
IP Family: IPv4
Topology: L3 Outbound
Click Save & Next

SSL Configuration

Create a New SSL Profile
Client-side SSL (Cipher Type): Cipher String
Client-side SSL (Cipher String): DEFAULT
Client-side SSL (Certificate Key Chain): default.crt and default.key
Client-side SSL (CA Certificate Key Chain): subca.f5demolabs.com
Server-side SSL (Cipher Type): Cipher String
Server-side SSL (Cipher String): DEFAULT
Server-side SSL (Trusted Certificate Authority): ca-bundle.crt
Click Save & Next

Service List

Inline Layer 2 service
1. Name: some name (ex. FireEye)
2. Network Configuration
  - Ratio: 1
  - From BIGIP VLAN: Create New, name (ex. FireEye_in), int 1.6
  - To BIGIP VLAN: Create New, name (ex. FireEye_out), int 1.7
  - Click Done
3. Service Action Down: Ignore
4. Enable Port Remap: Enable, 8080
5. Click Save
Inline layer 3 service
1. Name: some name (ex. IPS)
2. IP Family: IPv4
3. Auto Manage: Enabled
4. To Service Configuration
  - To Service: 198.19.64.7/25
  - VLAN: Create New, name (ex. IPS_in), interface 1.3, tag 50
5. Service Action Down: Ignore
6. L3 Devices: 198.19.64.64
7. From Service Configuration
  - From Service: 198.19.64.245/25
  - VLAN: Create New, name (ex. IP_out), interface 1.3, tag 60
8. Enable Port Remap: Enabled, 8181
9. Manage SNAT Settings: None
10. Click Save
Inline HTTP service
1. Name: some name (ex. Proxy)
2. IP Family: IPv4
3. Auto Manage: Enabled
4. Proxy Type: Explicit
5. To Service Configuration
  - To Service: 198.19.96.7/25
  - VLAN: Create New, name (ex. Proxy_in), interface 1.3, tag 110
6. Service Action Down: Ignore
7. HTTP Proxy Devices: 198.19.96.66, Port 3128
8. From Service Configuration
  - From Service: 198.19.96.245/25
  - VLAN: Create New, name (ex. Proxy_out), interface 1.3, tag 120
9. Manage SNAT Settings: None
10. Authentication Offload: Disabled
11. Click Save
ICAP Service
1. name: some name (ex. DLP)
2. IP Family: IPv4
3. ICAP Devices: 10.70.0.10, Port 1344
4. Request URI Path: /squidclamav
5. Response URI Path: /squidclamav
6. Preview Max Length(bytes): 524288
7. Service Action Down: Ignore
8. Click Save
TAP Service
1. Some Name (ex. TAP)
2. Mac Address: 12:12:12:12:12:12
3. VLAN: Create New, name (ex. TAP_in)
4. Interface: 1.4
5. Service Action Down: Ignore
6. Click Save
7. Click Save & Next

Service Chain List

Add
1. Name: some name (ex. my-service-chain)
2. Services: all of the services
3. Click Save
Add
1. name: some name (ex. my-sub-service-chain)
2. Services: L2 and TAP services
3. Click Save
Click Save & Next

Security Policy

Add a new rule
1. Name: some name (ex. urlf_bypass)
2. Conditions
  - Category Lookup (All)
  - SNI Category: Financial Data and Services, Health and Medicine
3. Action: Allow
4. SSL Forward Proxy Action: bypass
5. Service Chain: L2/TAP service chain
6. Click OK
Modify the All rule
1. Service Chain: all services chain
2. Click OK
Click Save & Next

Interception Rule

Select Outbound Rule Type: Default
Ingress Network (VLANs): client-side
L7 Interception Rules: apply FTP and email protocols as required
Click Save & Next

Egress Setting

Manage SNAT Settings: Auto Map
Gateways: New, ratio 1, 10.30.0.1

Summary

Review configuration
Click Deploy

2.2.8.2. Lab 2 demo script¶

Configuration review and prerequisites

Optionally define DNS, NTP and gateway route
Click Next

Topology Properties

Name: some name (ex. sslo-inbound-1)
Protocol: TCP
IP Family: IPv4
Topology: L3 Inbound
Click Save & Next

SSL Configuration

Show Advanced Setting
Client-side SSL (Cipher Type): Cipher String
Client-side SSL (Cipher String): DEFAULT
Client-side SSL (Certificate Key Chain): default.crt and default.key
Server-side SSL (Cipher Type): Cipher String
Server-side SSL (Cipher String): DEFAULT
Server-side SSL (Trusted Certificate Authority): ca-bundle.crt
Advanced (Expire Certificate Control): Ignore
Advanced (Untrusted Certificate Authority): Ignore
Click Save & Next

Services List

Click Save & Next

Service Chain List

Click Save & Next

Security Policy

Remove Pinners_Rule
Edit All Traffic rule and add L2/TAP service chain
Click Save & Next

Interception Rule

Gateway-mode
1. Hide Advanced Setting
2. Source Address: 0.0.0.0/0
3. Destination Address/Mask: 0.0.0.0/0
4. Port: 443
5. VLANs: outbound
Targeted-mode
1. Show Advanced Setting
2. Source Address: 0.0.0.0/0
3. Destination Address: 10.30.0.200
4. Port: 443
5. VLANs: outbound
6. Pool: webserver-pool
Click Save & Next

Egress Settings

Manage SNAT Settings: Auto Map
Gateways: Default Route

Summary

Review configuration
Click Deploy

2.2.8.3. Lab 3 demo script¶

Configuration review and prerequisites

Optionally define DNS, NTP and gateway route
Click Next

Topology Properties

Name: some name (ex. sslo-explicit)
Protocol: TCP
IP Family: IPv4
Topology: L3 Explicit Proxy
Click Save & Next

SSL Configuration

SSL Profile: Use Existing, existing outbound SSL settings
Click Save & Next

Services List

Click Save & Next

Service Chain List

Click Save & Next

Security Policy

Type: Use Existing, existing outbound security policy
Click Save & Next

Interception Rule

IPV4 Address: 10.20.0.150
Port: 3128
VLANs: client-net
Click Save & Next

Egress Settings

Manage SNAT Settings: Auto Map
Gateways: Existing Gateway Pool, -ex-pool-4 pool

Summary

Review configuration
Click Deploy

System Settings

DNS Query Resolution: Local Forwarding Nameserver
Local Forwarding Nameserver(s): 10.1.20.1
Click Deploy