Setting up a HA Galera Cluster using MaxScale

tullyLeave a comment

This is a guide meant to provide you with a simple, highly available, and balanced mysql cluster that works and is resilient as hell.

First, you’ll need five machines or five VM’s. I’m running Ubuntu 14.04 so to follow the guide you should as well. I’ll be notating each galera box as node1-3 and each maxscale box as ha1-2.

We need to add the galera repository. SSH into all three nodes and run the following

root@galera01:~# sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys cbcb082a1bb943db
root@galera01:~# echo 'deb http://mirror.jmu.edu/pub/mariadb/repo/5.5/ubuntu trusty main' | sudo tee /etc/apt/sources.list.d/galera.list
root@galera01:~# sudo apt-get update

Once that’s done we need to install galera and set the root password

root@galera01:~# sudo apt-get install mariadb-galera-server galera

If the service started automatically make sure and stop it now on all nodes. Otherwise what we do in later steps will require you to force it to die. 

root@galera01:~# sudo service mysql stop

Next edit /etc/mysql/my.cnf. I’m going to set the bind address to 0.0.0.0 so it’ll work for anyone but you can lock it down to a specific IP if you want. We can’t use loopback because the cluster members have to talk to one another. I’m also going to enable the slow log. 

bind-address		= 0.0.0.0
slow_query_log=1

Now, we need to create /etc/mysql/conf.d/cluster.cnf. The important variables in the file are the bind-address which once again is set to 0.0.0.0 for any IP. The wsrep_cluster_name can be anything you’d like but should be the same on all nodes of the cluster. The wsrep_cluster_address field should contain IP’s for all 3 of your nodes as shown. The wsrep_sst_method I use is rsync. Simply put, the gcache size is a cache of replication actions that need to be performed on a node that goes down temporarily and this will function over IST. If you go over your gcache you’ll use SST and rsync. This will lock the donor node until the failed node comes back up. You could use mysqldump but it’s slower. Lastly, your wsrep_node_address should be the node’s address you’re editing the file on. The same goes for the wsrep_node_name. I’ve added the last line wsrep_retry_autocommit because there are occasions where a cluster wide deadlock happens when another node is writing to a table that another node is also trying to write to. These only apply to autocommit statements. I chose to include this because I have seen occurrences where an application encounters a deadlock and the driver, despite being told “retry the transation”, simply errors out. 

[mysqld]
query_cache_size=0
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_type=0
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
#wsrep_provider_options="gcache.size=32G"

# Galera Cluster Configuration
wsrep_cluster_name="my_cluster"
wsrep_cluster_address="gcomm://192.168.1.130,gcomm://192.168.1.198,gcomm://192.168.1.167"

# Galera Synchronization Congifuration
wsrep_sst_method=rsync
wsrep_sst_auth=galerauser:secretsdontmakefriends

# Galera Node Configuration
wsrep_node_address="192.168.1.130"
wsrep_node_name="galera01"
wsrep_retry_autocommit=4

Next, we need to copy /etc/mysql/debian.cnf from the first node and replace the contents of the file with every other node. You can simply copy/paste or scp the file over. It says in caps “DO NOT TOUCH” but I’m an adult and can do what I want. The main piece we need syncd is the password= field. We’ll bootstrap every other node off the first so it will work fine. Don’t copy and paste mine below, I’m simply including it so you can reference what it looks like.

# DO NOT COPY THIS - USE THE ONE ON YOUR FIRST NODE
#Automatically generated for Debian scripts. DO NOT TOUCH!
[client]
host     = localhost
user     = debian-sys-maint
password = QLlRzgOp4ZIDVSVF
socket   = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host     = localhost
user     = debian-sys-maint
password = QLlRzgOp4ZIDVSVF
socket   = /var/run/mysqld/mysqld.sock
basedir  = /usr

Create a .my.cnf file in /root if you don’t feel like typing passwords

[client]
user=root
password="bananah4mmock"

[mysql]
user=root
password="bananah4mmock"

Run mysql_secure_installation on the first node. 

root@galera01:~# mysql_secure_installation 

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

You already have a root password set, so you can safely answer 'n'.

Change the root password? [Y/n] n
 ... skipping.

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] 
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] 
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] 
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] 
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Stop mysql on all nodes. Then, on the first node we’ll start our new cluster.

root@galera01:~# service mysql stop
 * Stopping MariaDB database server mysqld                    [ OK ] 
root@galera01:~# service mysql start --wsrep-new-cluster
 * Starting MariaDB database server mysqld                    [ OK ]

Now, on the other nodes we will simply start the mysql service. It’ll bootstrap off the first automatically. 

root@galera02:~# service mysql stop
 * Stopping MariaDB database server mysqld                    [ OK ] 
root@galera02:~# service mysql start
 * Starting MariaDB database server mysqld                    [ OK ]

Now, since we were lazy and created a .my.cnf file just run mysql -u root and get dropped into a shell. We’re going to create a database and watch it get replicated to the other nodes

root@galera01:~# mysql -u root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 30
Server version: 5.5.51-MariaDB-1~trusty-wsrep mariadb.org binary distribution, wsrep_25.14.r9949137

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database replicateme;
Query OK, 1 row affected (0.00 sec)

Head over to another node and log into mysql and run a show databases. You’ll see that our database was replicated just fine.

root@galera02:~# mysql -u root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 31
Server version: 5.5.51-MariaDB-1~trusty-wsrep mariadb.org binary distribution, wsrep_25.14.r9949137

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| replicateme        |
+--------------------+
4 rows in set (0.00 sec)

Now, you’re free to use this cluster. If you follow all the galera guidelines and don’t do anything stupid ever then you can use this as a master-master-master without problem using a fancy F5 or haproxy in a round robin. However, people and programs aren’t perfect and rewriting everyone’s code isn’t a viable solution. So, to eliminate deadlocks entirely, monitor the cluster, and have automatic failover we’re going to set up MaxScale. We are going to set up MaxScale to analyze the galera cluster, elect a master, analyze all incoming queries, and split writes to the master and reads to the rest of the slaves. Galera uses optimistic cluster wide locking. So, if you update a row on one node and do it on another node at the same time you can run into a deadlock. Splitting the queries will eliminate this.

To get all the fancy resiliency this post is meant to illustrate to do we need to add the maxscale repo, update apt, and install maxscale and keepalived. Do this on both nodes.

root@ha01:~# sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 70E4618A8167EE24
root@ha01:~# echo 'deb https://downloads.mariadb.com/files/MaxScale/1.4.3/ubuntu trusty main' |sudo tee /etc/apt/sources.list.d/maxscale.list
root@ha01:~# apt-get update
root@ha01:~# apt-get install maxscale keepalived

We’ll set up keepalived first. Edit /etc/keepalived/keepalived.conf on the first ha node. We’re going to make this the master. It’s priority needs to be higher than the second ha box. Your virtual_ipaddress needs to be an IP we haven’t used before and isn’t in use by any other system. Keepalived is going to claim it. The track script simply checks to see if maxscale is running and fails over to the slave if it isn’t. 

vrrp_script chk_maxscale
{
	script "killall -0 maxscale"
	interval 2
	weight 2
}

vrrp_instance VI_1
{
	interface eth0
	state MASTER
	virtual_router_id 51
	priority 101
	virtual_ipaddress
	{
		192.168.1.232
	}
	track_script
	{
		chk_maxscale
	}
}

Now, on our second ha box we’ll create /etc/keepalived/keepalived.conf again but set it up to be the failover

vrrp_script chk_maxscale
{
	script "killall -0 maxscale"
	interval 2
	weight 2
}

vrrp_instance VI_1
{
	interface eth0
	state BACKUP
	virtual_router_id 51
	priority 100
	virtual_ipaddress
	{
		192.168.1.232
	}
	track_script
	{
		chk_maxscale
	}
}

With that done we can edit /etc/sysctl.conf and add a line allowing us to bind services to IP’s we don’t really own. Add this to the bottom on both nodes

net.ipv4.ip_nonlocal_bind = 1

Load in the sysctl settings

root@ha01:~# sysctl -p
net.ipv4.ip_nonlocal_bind = 1

Now, we’ll set up maxscale. First we need to generate a .secret file. Do this with maxkeys.

root@ha01:~# maxkeys 
Generating .secrets file in /var/lib/maxscale ...

Next hash and remember a password we’ll use in a second for maxscale to monitor the cluster. You need to run this on both nodes and copy the hashes or run it on one and copy the .secrets file it creates to the second.

root@ha01:~# maxpasswd /var/lib/maxscale/ superduperpassword
F60FB938B468277026925BBEBCD35E5ACBA982A1AD2A9DA2956834CAE12C996C
2016-08-25 04:53:25   notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.

On the first galera node we’re going to create a user and grant it some permissions. I’m using wildcards but feel free to relegate it to your ha boxes’ main IP’s. Remember that these changes will now replicate so you only have to do it once.

MariaDB [(none)]> CREATE USER 'maxscale'@'%' IDENTIFIED BY 'superduperpassword';
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> GRANT SELECT ON mysql.user TO 'maxscale'@'%'; 
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> GRANT SELECT ON mysql.db TO 'maxscale'@'%';
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> GRANT SELECT ON mysql.tables_priv TO 'maxscale'@'%';
Query OK, 0 rows affected (0.02 sec)

MariaDB [(none)]> GRANT SHOW DATABASES ON *.* TO 'maxscale'@'%';
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> GRANT REPLICATION CLIENT ON *.* TO 'maxscale'@'%';
Query OK, 0 rows affected (0.00 sec)

Next, we need to edit /etc/maxscale.cnf on both ha boxes. Your config should match mine so remove any extra config info from the file. The threads should be set to your CPU count. The servers in the Splitter Service should be the names in [] that define your galera nodes. Mine are galera1-3. The user is the user you created in the previous step and the passwd field is the password hash that was output when you ran maxpasswd. In the Splitter Listener you need to define the virtual IP you set in keepalived.conf. Mine was 192.168.1.232. On each of the [galera#] sections you need to specify the node’s IP and mysql port. The Galera Monitor section should mirror the Splitter Service section in relation to the servers, user, and passwd. Leave the CLI sections alone. Remember that if you ran maxpasswd on both nodes instead of copying the .secrets file that you will have different hashes between the two servers.

[maxscale]
threads=4

[Splitter Service]
type=service
router=readwritesplit
servers=galera1, galera2, galera3
user=maxscale
passwd=F60FB938B468277026925BBEBCD35E5ACBA982A1AD2A9DA2956834CAE12C996C

[Splitter Listener]
type=listener
service=Splitter Service
protocol=MySQLClient
port=3306
address=192.168.1.232
socket=/tmp/ClusterMaster

[galera1]
type=server
address=192.168.1.226
port=3306
protocol=MySQLBackend

[galera2]
type=server
address=192.168.1.176
port=3306
protocol=MySQLBackend

[galera3]
type=server
address=192.168.1.163
port=3306
protocol=MySQLBackend

[Galera Monitor]
type=monitor
module=galeramon
servers=galera1, galera2, galera3
user=maxscale
passwd=F60FB938B468277026925BBEBCD35E5ACBA982A1AD2A9DA2956834CAE12C996C

[CLI]
type=service
router=cli

[CLI Listener]
type=listener
service=CLI
protocol=maxscaled
port=6603

That’s it for the configuration. Now, let’s start keepalived. Once you do you should see your virtual IP appear. Do this on both ha boxes.

root@ha01:~# service keepalived start
 * Starting keepalived keepalived                           [ OK ] 
root@ha01:~# ip addr show dev eth0
41: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:12:98:dc:0f:d0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.216/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.232/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6012:98ff:fedc:fd0/64 scope link 
       valid_lft forever preferred_lft forever

Once that’s done we can finally start maxscale and see if we screwed up. 

root@ha01:~# service maxscale start
 * Starting MaxScale                                                                                                                                                                                                                                                                       * maxscale is running
                                                                                                                                                                                                                                                                                   [ OK ]
root@ha01:~# cat /var/log/maxscale/maxscale1.log 


MariaDB Corporation MaxScale	/var/log/maxscale/maxscale1.log Thu Aug 25 05:13:00 2016
-----------------------------------------------------------------------
2016-08-25 05:13:00   notice : Configuration file: /etc/maxscale.cnf
2016-08-25 05:13:00   notice : Log directory: /var/log/maxscale
2016-08-25 05:13:00   notice : Data directory: /var/lib/maxscale
2016-08-25 05:13:00   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
2016-08-25 05:13:00   notice : Service cache: /var/cache/maxscale
2016-08-25 05:13:00   notice : Initialise CLI router module V1.0.0.
2016-08-25 05:13:00   notice : Loaded module cli: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libcli.so
2016-08-25 05:13:00   notice : Initializing statemend-based read/write split router module.
2016-08-25 05:13:00   notice : Loaded module readwritesplit: V1.0.2 from /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so
2016-08-25 05:13:00   notice : Initialise the MySQL Galera Monitor module V2.0.0.
2016-08-25 05:13:00   notice : Loaded module galeramon: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libgaleramon.so
2016-08-25 05:13:00   notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
2016-08-25 05:13:00   notice : No query classifier specified, using default 'qc_mysqlembedded'.
2016-08-25 05:13:00   notice : Loaded module qc_mysqlembedded: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libqc_mysqlembedded.so
2016-08-25 05:13:00   notice : qc_mysqlembedded loaded.
2016-08-25 05:13:01   notice : Query classifier initialized.
2016-08-25 05:13:01   notice : MariaDB Corporation MaxScale 1.4.3 (C) MariaDB Corporation Ab 2013-2015
2016-08-25 05:13:01   notice : MaxScale is running in process 4000
2016-08-25 05:13:01   notice : Loaded 2 MySQL Users for service [Splitter Service].
2016-08-25 05:13:01   notice : Loaded module MySQLClient: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libMySQLClient.so
2016-08-25 05:13:01   notice : Listening MySQL connections at 192.168.1.232:3306
2016-08-25 05:13:01   notice : Listening MySQL connections at /tmp/ClusterMaster
2016-08-25 05:13:01   notice : Loaded module maxscaled: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmaxscaled.so
2016-08-25 05:13:01   notice : Listening maxscale connections at 0.0.0.0:6603
2016-08-25 05:13:01   notice : Started MaxScale log flusher.
2016-08-25 05:13:01   notice : MaxScale started with 4 server threads.
2016-08-25 05:13:01   notice : Server changed state: galera1[192.168.1.226:3306]: new_slave
2016-08-25 05:13:01   notice : Server changed state: galera2[192.168.1.176:3306]: new_master
2016-08-25 05:13:01   notice : Server changed state: galera3[192.168.1.163:3306]: new_slave

Let’s log into maxscale and see if it worked. The default password is mariadb. If you create a new user with “add user username password” it will disable the normal login and you can log in using “maxscale -u username”

root@ha01:~# maxadmin
Password: 
MaxScale> list servers
Servers.
-------------------+-----------------+-------+-------------+--------------------
Server             | Address         | Port  | Connections | Status              
-------------------+-----------------+-------+-------------+--------------------
galera1            | 192.168.1.226   |  3306 |           0 | Slave, Synced, Running
galera2            | 192.168.1.176   |  3306 |           0 | Master, Synced, Running
galera3            | 192.168.1.163   |  3306 |           0 | Slave, Synced, Running
-------------------+-----------------+-------+-------------+--------------------

Let’s test failover now. On the first ha box show the existing IP’s, stop maxscale, wait 2 seconds, and check the IP’s again.

root@ha01:~# ip addr show dev eth0
41: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:12:98:dc:0f:d0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.216/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.232/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6012:98ff:fedc:fd0/64 scope link 
       valid_lft forever preferred_lft forever
root@ha01:~# service maxscale stop
 * Stopping MaxScale                                                 [ OK ]
root@ha01:~# ip addr show dev eth0
41: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:12:98:dc:0f:d0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.216/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6012:98ff:fedc:fd0/64 scope link 
       valid_lft forever preferred_lft forever

After stopping maxscale we lost our 232 IP. Switch to the second ha box and list it’s IP’s. You’ll see that it now exists there.

root@ha02:~# ip addr show dev eth0
43: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:8e:c4:53:4c:0b brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.114/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.232/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::148e:c4ff:fe53:4c0b/64 scope link 
       valid_lft forever preferred_lft forever

Last step is to turn maxscale back on on the first node and ensure we get our IP back.

root@ha01:~# service maxscale start
 * Starting MaxScale                                                                                                                                                                                                                                                                       * maxscale is running                                                 [ OK ]
root@ha01:~# ip addr show dev eth0
41: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:12:98:dc:0f:d0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.216/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.232/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6012:98ff:fedc:fd0/64 scope link 
       valid_lft forever preferred_lft forever

All you have to do now is set maxscale to autostart and you’re done. 

root@ha01:~# update-rc.d maxscale defaults
 Adding system startup for /etc/init.d/maxscale ...
   /etc/rc0.d/K20maxscale -> ../init.d/maxscale
   /etc/rc1.d/K20maxscale -> ../init.d/maxscale
   /etc/rc6.d/K20maxscale -> ../init.d/maxscale
   /etc/rc2.d/S20maxscale -> ../init.d/maxscale
   /etc/rc3.d/S20maxscale -> ../init.d/maxscale
   /etc/rc4.d/S20maxscale -> ../init.d/maxscale
   /etc/rc5.d/S20maxscale -> ../init.d/maxscale

Congrats, you just set up a bad ass resilient galera cluster. Set all your clients to hit your new virtual IP and rejoice in the knowledge that the only thing that can stops you now is someone dropping an entire database and it replicating to your other nodes… But you have backups for that right?

Spamming With Asterisk

tully2 Comments

I wrote this simple script years ago when I needed to load test a bunch of Asterisk machines and simply scaled it back down to a level that’s simply annoying. Be aware that this can completely tie up every line you have if you set it to. I don’t advocate using this for malicious purposes. You could cron it to set a reminder or something less nefarious. To use this you need to have followed my previous guide on how to set up Asterisk. The code and functionality is pretty simple. 

#!/usr/bin/perl

$numbertocall = "15552223333";
$numberofcalls = "200";

for($j=0;$j<$numberofcalls;$j++)
{
        $first = int(rand(899) + 100);
        $second = int(rand(899) + 100);
        $last = int(rand(8999) + 1000);
        $callerid = $first.$second.$last;
        open($fh, '>', "/tmp/$number.call");
        print $fh "Channel: SIP/flowroute/$numbertocall\n";
        print $fh "Application: Playback\n";
        print $fh "Data: tt-weasels\n";
        print $fh "CallerID: $callerid\n";
        close $fh;
        system("chown asterisk /tmp/$number.call");
        system("mv /tmp/$number.call /var/spool/asterisk/outgoing/");
        print "Called $numbertocall using $first-$second-$last\n";
        sleep(30);
}

Let’s go over the code.

This should be fairly self evident. You plug in the number to dial and how many time it’s supposed to dial it. It’s currently set to 200. Note that if you keep this set at 200 and ignore the resulting calls that it will leave a voicemail saying “weasels have eaten your phone system” from a random number 200 times. 

$numbertocall = "15552223333";
$numberofcalls = "200";

Our for loop simply keeps creating calls until the limit is reached or you ctrl+c

for($j=0;$j<$numberofcalls;$j++)
{
...
}

Here’s the fun part. We generate a random caller-id that will get set and shown on the recipient’s phone. Every time. 

$first = int(rand(899) + 100);
$second = int(rand(899) + 100);
$last = int(rand(8999) + 1000);
$callerid = $first.$second.$last;

Now, we write a file to /tmp containing the actual call information. This includes using flowroute as our outbound SIP trunk, the built in Asterisk application called Playback, the included sound file called “tt-weasels”, and we set our caller-id to what we generated earlier. 

open($fh, '>', "/tmp/$number.call");
print $fh "Channel: SIP/flowroute/$numbertocall\n";
print $fh "Application: Playback\n";
print $fh "Data: tt-weasels\n";
print $fh "CallerID: $callerid\n";
close $fh;

Next, we chown the file so that the asterisk user can manage it and move it to “/var/spool/asterisk/outgoing”. This folder is special. Asterisk is constantly watching it for .call files like we’re generating and automatically picks them up and spawns a call with the contained information. You could achieve something similar using the AMI but this works perfectly fine as well. Once the call is moved we sleep for 30 seconds before generating another. 

system("chown asterisk /tmp/$number.call");
system("mv /tmp/$number.call /var/spool/asterisk/outgoing/");
print "Called $numbertocall using $first-$second-$last\n";
sleep(30);

Assuming you named the file “dialer.pl” you can simply run “perl dialer.pl” to start it. Output should look similar to this

root@asterisk:~# perl dialer.pl 
Called 15552223333 using 555-111-4444

If you’d like to watch the calls being picked up and processed by asterisk you can run “asterisk -rvvv” to get verbose output

asterisk*CLI> 
    -- Attempting call on SIP/flowroute/15552223333 for application Playback(tt-weasels) (Retry 1)
  == Using SIP RTP CoS mark 5
    -- <SIP/flowroute-00000004> Playing 'tt-weasels.gsm' (language 'en')
[Feb 24 00:30:44] NOTICE[1365]: pbx_spool.c:402 attempt_thread: Call completed to SIP/flowroute/15552223333
asterisk*CLI>

Use this responsibly and don’t encapsulate the existing for loop with another to generate a lot of concurrent calls to completely tie up the phone system of telemarketers forcing them to change their DID blocks.

Asterisk Setup

tullyLeave a comment

I’ve been using Asterisk off and on for the past 10 years or so. For those wondering it’s basically software phone system often called a PBX. You can use it to do some very crazy things. I won’t cover those in this post as it’s meant to simply be a precursor to assist in setting it up for later posts. You’ll need a VPS or a Linux VM. I’ll be doing this on Ubuntu 14.04 but any debian derivative, Arch, and most other distros should have it in the repo.

First, ssh into your machine and install Asterisk.

root@asterisk:~# apt-get update && apt-get install asterisk

Once that’s done we need a way to actually make phone calls. I use a SIP provider called FlowRoute found at flowroute.com. I’ve used them for several years now with no issues whatsoever. The prices are cheap ($0.0098/min for US-48) so credits usually last me a while. They’ll also give you $0.25 to start with so you can try it out. Create an account there and log in. Once you’re in you need to click the “Interconnection” button at the top.

interconnection

From here you need to click the “System Configurator” link in orange underneath the “Tech Prefix” section

configurator

Then select “Asterisk” from the dropdown menu

configuration

Once you’re there all you should have to do is copy and paste the values in the three black boxes and put them at the end of their respective files. 

/etc/asterisk/sip.conf

sip.conf

/etc/asterisk/extensions.conf

extensions.conf

Once done check to make sure Asterisk is running and start it if it’s not.

root@asterisk:~# service asterisk status
 * Asterisk PBX is running

Then open the Asterisk shell with “asterisk -r” and reload the config with “reload”

root@asterisk:~# asterisk -r
Privilege escalation protection disabled!
See https://wiki.asterisk.org/wiki/x/1gKfAQ for more details.
Asterisk 11.7.0~dfsg-1ubuntu1, Copyright (C) 1999 - 2013 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 11.7.0~dfsg-1ubuntu1 currently running on asterisk (pid = 942)
asterisk*CLI> reload
asterisk*CLI>

Then you should be able to run “sip show peers” and show a connection to flowroute

asterisk*CLI> sip show peers
Name/username             Host                                    Dyn Forcerport ACL Port     Status      Description                      
flowroute/redacted        333.333.333.333                               a             5060     Unmonitored                                  
1 sip peers [Monitored: 0 online, 0 offline Unmonitored: 1 online, 0 offline]

That’s it! You’re all set up. One thing you should absolutely do at this point is secure your Asterisk installation. If you’re on a VPS you will have people try to brute force/scan your Asterisk service constantly. Read this page for information. I should have some new posts soon on fun things to do with Asterisk.

Open Plex Servers

tully8 Comments

I was having a bit of an issue with Plex the other day with it being accessible without using my VPN and got it working by opening the default 32400 port. I didn’t leave it like that but I was wondering who did. It turns out that a little over 3,000 people did. Pretty much any of those IP’s will give you full access to their library if you simply add /web to the end of the URL that shodan returns. What’s worse is that you can then queue up jobs to re-encode all the video or simply delete the libraries all together. I don’t advise this and simply verified that functionality on my already installed Plex server before fixing my own issue.

openplex

 

Search pattern is here. You’ll need to log in to view it.

https://www.shodan.io/search?query=X-Plex-Protocol+-unauthorized

GPU Passthrough Guide

tully68 Comments

This guide covers how to set GPU passthrough using Arch and Nvidia. I originally wrote this guide on reddit but decided to put it here in case that one gets removed.

First, I’d like to show you the results of this guide. Here’s a firestrike run using gpu passthrough.

This is meant to be a start to finish, holy shit this actually works, guide and is another lengthy post because there’s a lot to cover so stick with it and you’ll be happy you did. This post is going to cover UEFI specific hardware because every GPU made in the last few years has had it. I believe Nvidia implemented UEFI bios in the 600 series cards and some of you may need to flash the bios for that support, so all of you 500 series owners looking to pass them through will need to refer to the previous post and you want the q35 bios method. I’m unsure on when exactly AMD implemented support so do your research, I’m also supporting Intel/Nvidia exclusively in this post because I don’t own any AMD hardware. Everyone else can continue reading.

My Hardware Config

CPU: i7 4790k

Our primary concern is VT-d support. This is our bread and butter tech that allows us to pass through the GPU.

Mobo: MSI z97a Gaming 7

It’s a bit of an upgrade for me because my g1.sniper h6 was giving me fits when I upgraded to 32GB of ram. Manual doesn’t say the MSI supports VT-d specifically but does say it supports “virtualization technology” which is what we’re after.

GPU: 1 x EVGA GTX 970 FTW, 1 x ASUS STRIX GTX 980ti

I got a fancy 1440p 144hz monitor and the 970 wasn’t cutting it so I picked up a 980ti. You do not need these expensive cards to achieve this. The 980ti supports UEFI. That’s all we’re after in this post. I’ve done this with everything from a gtx 260, 550ti, 970, and now 980ti.

Storage: 1 x Intel 730 240GB SSD, 4 x 1TB WD in Raid 10

I changed my storage up a bit for peace of mind reasons. I’m now running windows off of a qcow2 container rather than a physical hdd. You can do either one.

RAM: 32GB of ddr3 1600

Linux doesn’t require much ram. I simply have 32GB because I run a lot of VM’s for work and emulate an enterprise environment. Generally Linux doesn’t require much RAM and you can get by with 8-12GB easily.

Monitors: 2 x 1920×1080, 1 x 1440p

The gist of monitor setups with this is to wire everything twice. You need one cable going out of each GPU to each monitor. If your monitor only has one input you can buy a switch for whatever connection you need. If you get a switch you can either leave both outputs enabled all the time or turn the linux output off with xrandr and the switch should failover automatically to the second input. If you don’t have a switch then I would recommend using xrandr because it gets annoying to manually switch the inputs every time on the monitor.

Setup and Installation

I break things constantly on my system. So I’m using Antergos to install Arch. I put my /home on the RAID-10 and just change /etc/fstab if I need to reinstall so I never lose anything important. Pretty much any offshoot of Arch and vanilla Arch should work just fine for this post. I have also helped people do this on Debian and derivatives. For this guide we will be sticking to Arch. I’m using grub2 as my bootloader with UEFI.

The first thing we need to do is enable VT-d and ensure functionality. We need to edit /etc/default/grub . Look for the line that says “GRUB_CMDLINE_DEFAULT=”” and append “intel_iommu=on” to what is inside of that line. Mine looks like this

GRUB_CMDLINE_LINUX_DEFAULT="resume=UUID=6771936b-06b6-493c-b655-6f60122f5228 intel_iommu=on"

Once you’ve done that we need to rebuild our grub.cfg file. Run this to do so

sudo grub-mkconfig -o /boot/grub/grub.cfg

Next we reboot to activate iommu/vt-d. Once you’re back in Arch we need to verify that VT-d is enabled and functioning. First we need to identify the pci-e bus the GPU we’re passing through is on. We run “lspci -nnk” to find this information. Here are the lines important to me.


02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8548]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8548]
Kernel driver in use: snd_hda_in

My 980ti is in the second pci-e slot on my motherboard so this is correct. 02.00.1 is the audio bus for the card and is also important later on. Next we need to see if the 980ti falls into another pci-e bus’s iommu group and if that will conflict. To find this out you look in /sys/bus/pci/devices/YOUR_BUS/iommu_group/devices/. If you do not have an iommu_group folder then vt-d was not enabled properly! Here is the output for mine

 [kemmler@arch ~]$ ls -lha /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
 total 0
 drwxr-xr-x 2 root root 0 Sep 19 18:47 .
 drwxr-xr-x 3 root root 0 Sep 19 18:46 ..
 lrwxrwxrwx 1 root root 0 Sep 19 18:47 0000:00:01.0 -&gt; ../../../../devices/pci0000:00/0000:00:01.0
 lrwxrwxrwx 1 root root 0 Sep 19 18:47 0000:00:01.1 -&gt; ../../../../devices/pci0000:00/0000:00:01.1
 lrwxrwxrwx 1 root root 0 Sep 19 18:47 0000:01:00.0 -&gt; ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
 lrwxrwxrwx 1 root root 0 Sep 19 18:47 0000:01:00.1 -&gt; ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
 lrwxrwxrwx 1 root root 0 Sep 19 18:47 0000:02:00.0 -&gt; ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
 lrwxrwxrwx 1 root root 0 Sep 19 18:47 0000:02:00.1 -&gt; ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

0000:00:01.0 and 0000:00:01.1 can be ignored. The issue is 0000:01:00.0 and 0000:01:00.1. These are my 970 and will cause GPU passthrough to fail unless I pass both cards through to the VM. If you only see device of 0000:0#.00.# in your output then your iommu group is clean and you can skip this next section

Fixing IOMMU Grouping

I installed Antergos with AUR support. This provides a tool called yaourt. A very thoughtful member of the Arch community has been steadily providing a current kernel patched with a few fixes that deal with IOMMU groups and other issues. This package is called “linux-vfio”. Assuming you have yaourt installed you will run this.

yaourt -S linux-vfio

It will ask you if you want to edit a few things, you can say no. It will then ask you if you want to only install linux-vfio. Say NO so it will also install the docs and headers for the kernel. Proceed through the installation with common sense. Once you’ve installed it, rebuild your grub.cfg as above with “sudo grub-mkconfig -o /boot/grub/grub.cfg”.

Skip if using Nouveau

If you installed the “nvidia” package from the arch repo your driver will probably break in the new kernel. I simply installed the binary from nvidia’s website. A simple way to do that against the new kernel is to download the binary, reboot, edit grub by pressing “e” with linux-vfio selected, and then append “nomodeset systemd.unit=multi-user.target” to the end of the long line that begins with “linux …” so you do a one time edit of the boot parameters. Then navigate to the binary and “sh NVIDIA-###…sh” It should disable nouveau if needed and install. Reboot and continue. You may also want to look into nvidia-dkms if you plan on updating your linux-vfio kernel regularly.

Once you’re in the linux-vfio kernel you need to enable the acs_override patch. The easiest way is to just use the downstream method. There are other optionsbut should not be necessary. We will add “pcie_acs_override=downstream” to our grub.cfg at /etc/default/grub. “sudo grub-mkconfig -o /boot/grub/grub.cfg” once again to rebuild it.

GRUB_CMDLINE_LINUX_DEFAULT="resume=UUID=6771936b-06b6-493c-b655-6f60122f5228 pcie_acs_override=downstream intel_iommu=on"

Reboot and then check your iommu group again.


[kemmler@arch ~]$ ls -lha /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
drwxr-xr-x 2 root root 0 Sep 19 19:11 .
drwxr-xr-x 3 root root 0 Sep 19 19:11 ..
lrwxrwxrwx 1 root root 0 Sep 19 19:11 0000:02:00.0 -&gt; ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 Sep 19 19:11 0000:02:00.1 -&gt; ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

 

0000:01:00.0/1 are now missing from the initial output. This is exactly what we want to see. This means that the 980ti is now in it’s own IOMMU group and can be allocated to a VM by itself. We’re now ready to move on.

Setup Continued

The next thing we need to do is blacklist the GPU we’re passing through to the VM so that the Nvidia driver doesn’t try to grab it. Nvidia is a dick and doesn’t conform to standards properly. You can’t easily unbind a gpu from the Nvidia driver so we use a module called “pci-stub” to claim the card before nvidia can. We achieve this by putting pci-stub in our initramfs that is loaded before the kernel and passing a parameter to grub telling it what to do. So, edit “/etc/mkinitcpio.conf” and add “pci-stub” to the Modules=”” section like so

MODULES="pci-stub"

If you’re running the stock kernel use this to rebuild your initramfs

sudo mkinitcpio -p linux

linux-vfio users run this

sudo mkinitcpio -p linux-vfio

Now we edit grub once again and add our pci_stub options to bind our card to pci-stub. Get your device IDs from “lspci -nnk”. My id’s are “10de:17c8” and “10de:0fb0” as seen here again

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8548]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8548]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

Edit /etc/default/grub and then run “sudo grub-mkconfig -o /boot/grub/grub.cfg” Your line should similar to this. Remember that you need to pass what’s in the IOMMU group so you need the main card and the audio bus if it’s there.


GRUB_CMDLINE_LINUX_DEFAULT="resume=UUID=6771936b-06b6-493c-b655-6f60122f5228 pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:17c8,10de:0fb0"

Reboot. If everything goes to plan you should now see “pci-stub” as the module in use from your “lspci -nnk” output for the card.

Kernel driver in use: pci-stub

You’re all set. Now we can move on to installing the core software.

Setup KVM/QEMU

We need to install qemu first.

sudo pacman -S qemu

Next we need the UEFI bios called OVMF. Look hereand get the edk2.git-ovmf-x64-###.noarch.rpm file. Install rpmextract

sudo pacman -S rpmextract

Next we’ll extract the files and move them over as they had them.

 
[kemmler@arch ovmf]$ ls edk2.git-ovmf-x64-0-20150916.b1214.g2f667c5.noarch.rpm 
[kemmler@arch ovmf]$ rpmextract.sh edk2.git-ovmf-x64-0-20150916.b1214.g2f667c5.noarch.rpm 
[kemmler@arch ovmf]$ ls edk2.git-ovmf-x64-0-20150916.b1214.g2f667c5.noarch.rpm usr 
[kemmler@arch ovmf]$ sudo cp -R usr/share/* /usr/share/
[kemmler@arch ovmf]$ ls /usr/share/edk2.git/ovmf-x64/ 
OVMF_CODE-pure-efi.fd OVMF_CODE-with-csm.fd OVMF-pure-efi.fd OVMF_VARS-pure-efi.fd OVMF_VARS-with-csm.fd OVMF-with-csm.fd UefiShell.iso

Now we need to create our vfio-bind script that will replace our pci-stub placeholder driver. I suggest putting it in /usr/bin/vfio-bind and then running “chmod +x /usr/bin/vfio-bind”


#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Next we bind our gpu. Remember to bind the whole gpu if necessary which means both buses if present. Replace your pci bus

sudo vfio-bind 0000:0#:00.0 0000:0#:00.1

Verify that vfio-bind is now in control of the gpu with “lspci -nnk”

Kernel driver in use: vfio-pci

Now we can test it out and see if it works! Make sure to verify paths are correct, change your pci bus ID, and remove the second pci bus line if you only have one for your card. Throw this script in a file and run it like you did with the vfio-bind script. Once you do that you should be able to switch the input on your monitor or KVM switch and be greeted with a black terminal with yellow text. This is the UEFI shell and means that everything is working wonderfully!


#!/bin/bash

cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
qemu-system-x86_64 \
-enable-kvm \
-m 2048 \
-cpu host,kvm=off \
-vga none \
-device vfio-pci,host=02:00.0,multifunction=on \
-device vfio-pci,host=02:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd

Setting Up Windows

Now that we have video out all we need to do next is change our script up and install windows to a disk or to a qcow2 container. I’ll be doing the latter but can help with the former. You need a windows iso. I’m using windows 10 enterprise. You also need to get the VirtIO drivers from redhat. You can download them here. I am using the “Latest virtio-win iso” but stable should also be fine. First let’s create our qcow2 container. I’m using a few params to increase performance, if you have any tips on better methods I’d be glad to hear it. Command below. Change 120G to whatever size you want your container to be.

qemu-img create -f qcow2 -o preallocation=metadata,compat=1.1,lazy_refcounts=on win.img 120G

Next we modify our script to include the windows iso, virtio iso, and our new container. Once again, verify all path names. I’m also passing through my usb keyboard to the guest. This line begins with -usb. Find your usb device by using “lsusb”. I’d recommend not passing through your mouse just yet so that if you fuck up you can simply close the black qemu window that pops up to get your keyboard back. Alterantively just hook up a second keyboard if you have one. I always keep a spare around in case windows hangs. Notice I’m using writeback cache on my qcow2 image. Remove that if you do not want it.


#!/bin/bash

cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
qemu-system-x86_64 \
-enable-kvm \
-m 2048 \
-cpu host,kvm=off \
-vga none \
-usb -usbdevice host:1b1c:1b09 \
-device vfio-pci,host=02:00.0,multifunction=on \
-device vfio-pci,host=02:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/kemmler/kvm/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-drive file=/home/kemmler/kvm/win.img,id=disk,format=qcow2,if=none,cache=writeback -device scsi-hd,drive=disk \
-drive file=/home/kemmler/kvm/virt.iso,id=virtiocd,if=none,format=raw -device ide-cd,bus=ide.1,drive=virtiocd

Once you boot you should see the “press any key to boot from cd…” if you miss it you’ll eventually get dumped into a Shell> prompt. Just type “exit” hit enter, and navigate to “Boot Manager”. Select the first SCSI device and you should get the prompt again. Go through and install windows as you normally would. When you get to the disk selection screen it will prompt you for a driver. Navigate to the virtio iso, virtscsi folder, Windows 8.1, x64, assuming you’re using windows 10 x64 like me. Go through the rest of the install and then shut it down when done. Next we’re going to get sound working, add our mouse to the usb passthrough, and set our monitors to switch automatically with xrandr.

Run “xrandr” to find your output device names. They correspond to your connection types. eg dvi-d-0. I’m going to be using two monitors while in windows and leaving 1 for linux to keep conky running to monitor the system. My two monitors i’ll be switching are called “DVI-I-1” and “DVI-D-0”. I’m also changing the cpu values to match my 4790k and my ram to 8GB. Note the mode and the rate on the xrandr commands. This refers to the resolution and refresh rate respectively.


#!/bin/bash

xrandr --output DVI-I-1 --off
xrandr --output DVI-D-0 --off
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa
qemu-system-x86_64 \
-enable-kvm \
-m 8196 \
-smp cores=4,threads=2 \
-cpu host,kvm=off \
-vga none \
-soundhw hda \
-usb -usbdevice host:1b1c:1b09 -usbdevice host:046d:c07d \
-device vfio-pci,host=02:00.0,multifunction=on \
-device vfio-pci,host=02:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/kemmler/kvm/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-drive file=/home/kemmler/kvm/win.img,id=disk,format=qcow2,if=none,cache=writeback -device scsi-hd,drive=disk \
-drive file=/home/kemmler/kvm/virt.iso,id=virtiocd,if=none,format=raw -device ide-cd,bus=ide.1,drive=virtiocd
xrandr --output DVI-D-0 --mode "2560x1440" --rate 60 --left-of HDMI-0
xrandr --output DVI-I-1 --mode "1920x1080" --rate 144 --left-of DVI-D-0

From here you should be able to install the nvidia driver, steam, etc. You’ll probably need to reboot to get the nvidia driver working. Once that’s done everything should be great.

Common Questions/Problems and Final Thoughts

  1. Can I SLI in the guest VM? Short answer is probably not. Neither of my lga 1150 motherboards will allow me to try.
  2. Can I use two identical GPUs with one being in each OS? You will run into problems binding just one with pci-stub if both gpus share the same identifier found in “lspci -nnk”. pci-stub will bind both of them and the nvidia driver will cease to function in linux. A potential workaround is to use xen’s pciback module. This will allow you to grab based on the pci bus rather than the device id but I haven’t tried this.
  3. Sound isn’t working! I’m using pulseaudio and the hda device. Honestly you’ll have to experiment. There’s a 200+ page forum post with people posting working configs here
  4. The guide is too long or not easy enough. This guide isn’t for you then.
  5. Can I use X device? Generally you can pass through any pci, or usb device. If you want an actual answer the only way would be to donate me the part so I can figure it out short of finding someone on google that claims it works.
  6. My windows ISO won’t boot! Make sure you’re using an unmodified copy. I’m not positive but a lot of the dual purpose ISOs and hand crafted don’t include bootx64.efi and I believe that’s the cause. I can confirm that a clean version of Windows 10 Enterprise x64 boots just fine.

Hopefully you found this guide useful. I’m sure it seems like a gigantic pain in the ass but really once you set it up you don’t have to mess with it again. I cannot stress how useful it is to simply be able to boot a VM play what I want and then turn it off without having to close all my shit by dual booting constantly. So let me know if this guide helped or how I can improve on it!