Rotating proxy pool for scraping

Well, it’s now legal to scrape LinkedIn.  Sometimes, though, as Lil Wayne has said, real g’s move in silence like lasagna.

One way to do this is to rotate one’s requests through a proxy.  Sometimes it’s rather convenient to just have one IP address to use as a proxy – and have that proxy forward through a round robin rotating pool.  For example, one could lease new proxy pools (via API or manually), without having to update the scraper code.

This is easy to do implement through Squid, an open source proxy.  Just provision an Ubuntu LTS instance, install squid + dependencies (there are many tutorials for Ubuntu LTS), and add this to the top of your squid config file:

# add this to the top of  /etc/squid/squid.conf
# then restart squid (service squid stop, service squid start)

request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Cache-Control allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access All deny all

via off
forwarded_for off
follow_x_forwarded_for deny all

acl deny24_25 random 24/25
acl deny23_24 random 23/24
acl deny22_23 random 22/23
acl deny21_22 random 21/22
acl deny20_21 random 20/21
acl deny19_20 random 19/20
acl deny18_19 random 18/19
acl deny17_18 random 17/18
acl deny16_17 random 16/17
acl deny15_16 random 15/16
acl deny14_15 random 14/15
acl deny13_14 random 13/14
acl deny12_13 random 12/13
acl deny11_12 random 11/12
acl deny10_11 random 10/11
acl deny9_10 random 9/10
acl deny8_9 random 8/9
acl deny7_8 random 7/8
acl deny6_7 random 6/7
acl deny5_6 random 5/6
acl deny4_5 random 4/5
acl deny3_4 random 3/4
acl deny2_3 random 2/3
acl deny1_2 random 1/2
acl deny0_1 random 1/1

never_direct allow all

cache_peer 206.66.98.244 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P25
cache_peer 206.66.98.234 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P24
cache_peer 206.66.98.223 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P23
cache_peer 206.66.98.217 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P22
cache_peer 206.66.98.211 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P21
cache_peer 206.66.98.188 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P20
cache_peer 206.66.98.148 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P19
cache_peer 206.66.98.123 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P18
cache_peer 206.66.98.119 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P17
cache_peer 206.66.98.110 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P16
cache_peer 206.66.98.105 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P15
cache_peer 206.66.98.99 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P14
cache_peer 206.66.98.84 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P13
cache_peer 206.66.98.83 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P12
cache_peer 206.66.98.71 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P11
cache_peer 206.66.98.63 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P10
cache_peer 206.66.98.42 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P9
cache_peer 206.66.98.34 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P8
cache_peer 206.66.98.14 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P7
cache_peer 206.66.98.200 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P6
cache_peer 206.66.98.152 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P5
cache_peer 206.66.98.120 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P4
cache_peer 206.66.98.56 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P3
cache_peer 206.66.98.32 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P2
cache_peer 206.66.98.2 parent 60099 0 no-query default login=zackX:CKtL6Iza name=P1

cache_peer_access P25 deny deny24_25
cache_peer_access P24 deny deny23_24
cache_peer_access P23 deny deny22_23
cache_peer_access P22 deny deny21_22
cache_peer_access P21 deny deny20_21
cache_peer_access P20 deny deny19_20
cache_peer_access P19 deny deny18_19
cache_peer_access P18 deny deny17_18
cache_peer_access P17 deny deny16_17
cache_peer_access P16 deny deny15_16
cache_peer_access P15 deny deny14_15
cache_peer_access P14 deny deny13_14
cache_peer_access P13 deny deny12_13
cache_peer_access P12 deny deny11_12
cache_peer_access P11 deny deny10_11
cache_peer_access P10 deny deny9_10
cache_peer_access P9 deny deny8_9
cache_peer_access P8 deny deny7_8
cache_peer_access P7 deny deny6_7
cache_peer_access P6 deny deny5_6
cache_peer_access P5 deny deny4_5
cache_peer_access P4 deny deny3_4
cache_peer_access P3 deny deny2_3
cache_peer_access P2 deny deny1_2
cache_peer_access P1 deny deny0_1

FYI, this setup is easy to test:

#!/usr/bin/env python3

import requests

url = "http://ip-api.com/json"
proxy = {"http": "http://droplet.ip.address:3128"}

r = requests.get(url, proxies=proxy)

print("Response:\n{}".format(r.text))

Just think: if one is programmatically (via API) leasing new proxy pools, one could easily write a script to procedurally generate this config file and restart this squid proxy…

Happy hacking 🙂

Free automatic GitHub backups with Keybase

Here’s how you setup Keybase to mirror (automatically backup) your GitHub repo:

  1. git remote set-url --add --push origin keybase://team/name/repo
  2. git remote set-url --add --push origin git@github.com:team/repo.git

The advantage is that if GitHub goes offline or decides to remove your data, your code is protected!

But it’s likely that not all committers have Keybase installed, and you don’t want to put an extra step or hurdle to getting your backups in order – introducing fragility, adding a point of failure.  In that case, how do you ensure that your hard work is seamlessly backed up?

Presenting a solution: https://github.com/codeforcash/github-to-keybase-mirror

This project deploys a serverless function and generates a webhook url that automatically pushes any commits to your Keybase git remote.

I had the good fortune of collaborating with a developer, Mark Koenig, who did the hard work of implementing the project.  He shared this writeup:

I learned a lot about Lambda functions and the AWS framework in general. I was surprised to see that when I tested locally the function would work perfectly, but once I deployed it to the AWS servers it would not work at all. I found that this is based on differences between the AWS sam-local Docker container and the production environment.

The sam-local container has different permissions for its file structure than the production Lambda containers. So instead of accessing the Keybase binaries right in their standard folder /var/task/gopath/bin I had to copy them to the /tmp folder so that I could update their permissions with chmod.

In the Lambda environment, you can only write to the /tmp folder, but Keybase kept trying to write to /home/user/... to make new directories and especially to make and access its keybased.sock file. This was compounded by the fact that if I changed the ‘HOME’ environment variable on the local Docker container, the function would not run at all! I was overjoyed when I found in the Keybase documentation that I could bypass both of these problems by instead adding a$XDG_RUNTIME_DIR environment variable and setting that to /tmp.  For more details on that, see the XDG Base Directory specification. Once I figured that out, it was pretty much smooth sailing for the rest.

All in all, even though it was challenging and frustrating at times, it was still pretty fun and very satisfying once it finally worked.

Thanks to dxb for feedback on this post.  Thanks to Alex Markley for pointing out that this is an easier, simpler solution that current techniques.

Announcing v0.2 of Croupier, the Keybase sweepstakes bot

TL;DR:  We are giving away XLM hourly in the @mkbot#cryptosnipe channel on Keybase

 

When we announced Croupier the Keybase bot, the project gained momentum.  Today we are announcing v0.2 of “DevCroupier”, with a laundry list of new features.  

Several gameplay mechanics were inspired by a game that I played in school, something perhaps described as pattycake-meets-Pokémon: players would clap their hands and then either “charge”, “attack”, or “block”.  

compressed-pe-game

Gameplay screenshots

croupier_v2_gameplay_-_s_1croupier_gameplay_v2_-_s_2

The rules are comprehensive and detail all the new features

Many people have asked: is this gambling?  We previously said: kinda, but technically, no. Now, we1 robustly say: no!  There are three major elements that make something gambling: prize, chance, and consideration for entry.  Croupier is a free promotional sweepstakes hosted by Code For Cash in order to demo features of the Keybase platform and attract applicants to jobs we’re recruiting for.  Important to note that:

  • No purchase required for entry – simply react in chat with a positive emoji
  • A skillful, persistent free player has a stronger chance of winning than a player with a high bankroll
  • The regulators’ guiding principle on whether to intercede is generally: was someone damaged?  We think the rules and gameplay are constructed in a way that prevents inadvertent harm
  • We tabulate people cumulative earnings and issue 1099s where appropriate – it’s true, check the source!
  • Pot sizes are capped at reasonable amounts (e.g. 20,000XLM) to avoid running into issues with bonding requirements for low-level sweepstakes
  • We take zero fee or rake

 

Croupier is still in active development.  We’re always testing gameplay, open to feedback and adding new features!  We invite you to join us in @mkbot on Keybase to enjoy a round of Cryptosnipe and maybe win some XLM – no Lumens necessary to play!

Footnotes

  1. If a regulator claims otherwise, that this is in violation of anything of consequence, our first order of business is going to be to shut the bot down immediately and milk the publicity for all it’s worth.

 

TL;DR:  We are giving away XLM hourly in the @mkbot#cryptosnipe channel on Keybase

Finding Alice and Bob in Wonderland: a writeup of Croupier, the Keybase bot

EDIT: version 0.2 of the bot is now released

 

You may have observed that plenty respectable, prolific Hacker News users0 have added a Keybase signature to their profile:

Hacker_News
Seems that 20%+ of the Hacker News leaderboard has a Keybase signature in their profile…

So. What is Keybase?

Keybase’s mission is to

make cryptographically secure communication, cryptocurrency, and identity management into a user-friendly experience

Well, that’s a tall order…1 

Anyway, Keybase is like Slack, except free.  

keybase_chat_screenshot____.png

And transparently encrypted.  And early stage software; all of the error handling is transparent.  And open source.  And the currently-small user population of ~300,000 is mostly programmers.  And the team is responsive to community requests2.

In March 2019, Keybase issued a pair of press releases that made it to the top of Hacker News. 

Please allow me to TL;DR these:

  • Users may now generate randomness, in a way that’s indisputably fair, within Keybase chat conversations.

    coin_animation2
  • Users may now send money in chat in a conversational way.
    keybase salad.png
  • The Stellar foundation, backed by some of the smartest people in tech including Patrick Collison and Keith Rabois, backs Keybase.  Sure, social proof matrices have their flaws, but if it worked for Google and PageRank…

Anyway, the sudden emergence of these two technologies – integrated moneysending and trustable randomness – led to a few synapses sparking and – as neural connections and memories tangled – sent me down a rabbit hole of nostalgia:

The year is 2002. I’m a pimply-faced youth attending high school, living most of my life in an online fantasy world, a MMORPG called Nexus: TK.

nexustk

My main character was a rogue, but I had the opportunity to specialize in the merchant subpath, and I learned a spell called ✨ gamble ✨.

Either player could ante up to 250,000 coins and then the winning player (presumably based on the outcome of a Random Number Generator) would be awarded the proceeds.  There would be a cool little animation around the winning character, so anyone could spectate the outcome of the gamble, celebrating with the winner and taking schadenfreudian delight in the loser’s defeat.

Anyway, it clicked to me that with the launch of Keybase’s new tech, I could implement the gamble spell within Keybase.  I decided to roll up my sleeves, and the result?

Croupier.

Keybase_2

Keybase_1

Keybase

Players start a round by sending croupier money in chat, with a minimum bet size of 0.01XLM:

+0.01XLM@croupier

The countdown clock begins, allowing people 60 seconds to join the betting round. After any contribution to the pot, the clock resets, giving people another 60 seconds to join.  Once the clock runs out, each player is assigned a numerical range proportional to the size of their bet. Keybase randomly generates a number and the proceeds (minus the transaction fees to receive the bets – extremely small, approximately 100 stroops or 0.000001XLM) are directed to whoever’s range contains the number.

Yes, this means that with a bet of just 0.01XLM, one could win a pot containing hundreds of lumens. Note: If the flip fails to resolve or there aren’t 2+ participants in the game, participants are automatically refunded.

Here’s the source code….

Build it with:

 tsc --lib es2015 index.ts

And then run it with:

node index.js
import * as _ from "lodash";
import * as mysql from "mysql";
import * as os from "os";
import * as Bot from "./keybase-bot";

import "source-map-support/register";

const bot: Bot = new Bot(os.homedir());
const bot2: Bot = new Bot(os.homedir());

const botUsername: string = "croupier";
const paperkey: string = process.env.CROUPIER_PAPERKEY_1;
const paperkey2: string = process.env.CROUPIER_PAPERKEY_2;

import { ChatChannel, MessageSummary, Transaction } from "./keybase-bot";

interface IParticipant {
  username: string;
  transaction: Transaction;
}

interface ISnipe {
  participants: Array;
  betting_open: boolean;
  clock: string;
  timeout: string;
}

function documentSnipe(channel: ChatChannel, winner: string, wasCancelled: boolean): void {
  const participants: string = JSON.stringify(activeSnipes[JSON.stringify(channel)].participants);
  const connection: mysql.Connection = mysql.createConnection({
    database : process.env.MYSQL_DB,
    host     : process.env.MYSQL_HOST,
    password : process.env.MYSQL_PASSWORD,
    user     : process.env.MYSQL_USER,
  });

  connection.connect();

  if (winner !== null) {
    winner = `'${winner}'`;
  }

  connection.query(`INSERT INTO snipes
    (participants, winner, was_cancelled)
    VALUES
    ('${participants}', ${winner}, ${wasCancelled})`, (error, results, fields) => {
      if (error) {
        console.log(error);
      }
    });

  connection.end();
}

function processRefund(txn: Transaction, channel: ChatChannel): void {
  console.log("refunding txn", txn);
  // API returns a response, number of stroops
  const transactionFees: number = 300 * 0.0000001;
  console.log("refunding txn fees", transactionFees);
  const refund: number = _.round(txn.amount - transactionFees, 7);
  console.log("total refund is", refund);
  bot.chat.sendMoneyInChat(channel.topicName, channel.name, refund.toString(), txn.fromUsername);
}

function extractTxn(msg: MessageSummary): void {
  const txnId: string = msg.content.text.payments[0].result.sent;
  bot.wallet.details(txnId).then((details) => processTxnDetails(details, msg.channel));
}

function sendAmountToWinner(winnerUsername: string, channel: ChatChannel): void {

  let bounty: number;
  const snipe: ISnipe = activeSnipes[JSON.stringify(channel)];

  bounty = 0;

  snipe.participants.forEach((participant) => {
     bounty += parseFloat(participant.transaction.amount);
     bounty -= (300 * 0.0000001); // transaction fees for receiving the transaction
  });

  bounty = _.round(bounty, 7);
  console.log("now rounded", bounty);

  bot.chat.sendMoneyInChat(channel.topicName, channel.name, bounty.toString(), winnerUsername);

}

function resolveFlip(channel: ChatChannel, winningNumber: number): void {

  let winnerUsername: string;
  const bettorRange: object = buildBettorRange(channel);
  Object.keys(bettorRange).forEach((username) => {
    if (bettorRange[username][0] <= winningNumber && bettorRange[username][1] >= winningNumber) {
      winnerUsername = username;
    }
  });

  sendAmountToWinner(winnerUsername, channel);
  bot.chat.send(channel, {
    body: `Congrats to @${winnerUsername}`,
  });

  documentSnipe(channel, winnerUsername, false);
}

function buildBettorRange(channel: ChatChannel): any {
  const bettorMap: object = {};
  activeSnipes[JSON.stringify(channel)].participants.forEach((participant) => {
    if (typeof(bettorMap[participant.username]) === "undefined") {
      bettorMap[participant.username] = Math.floor(participant.transaction.amount / 0.01);
    } else {
      bettorMap[participant.username] += Math.floor(participant.transaction.amount / 0.01);
    }
  });

  const bettorRange: object = {};
  let start: number = 0;
  Object.keys(bettorMap).forEach((key) => {
    bettorRange[key] = [start + 1, start + bettorMap[key]];
    start += bettorMap[key];
  });

  return bettorRange;
}

function flip(channel: ChatChannel): void {

  const bettorRange: object = buildBettorRange(channel);
  const bettingValues: Array = Object.values(bettorRange);
  const flatBettingValues: Array = _.flatten(bettingValues);

  const minBet: number = flatBettingValues.reduce((a, b) => Math.min(a, b));
  const maxBet: number = flatBettingValues.reduce((a, b) => Math.max(a, b));

  let bettingTable: string = "Betting table\n";

  Object.keys(bettorRange).forEach((username) => {
    bettingTable += `\n@${username}: \`${bettorRange[username][0]} - ${bettorRange[username][1]}\``;
  });

  bot2.chat.send(channel, {
    body: bettingTable,
  });
  bot2.chat.send(channel, {
    body: `/flip ${minBet}..${maxBet}`,
  });
}

function processTxnDetails(txn: Transaction, channel: ChatChannel): void {
  if (txn.toUsername !== botUsername) {
    return;
  }
  const isNative: boolean = txn.asset.type === "native";
  if (!isNative) {
    return;
  }
  if (parseFloat(txn.amount) < 0.01) {     bot.chat.send(channel, {       body: "Thanks for the tip, but bets should be >= 0.01XLM",
    });
    return;
  }

  const snipe: ISnipe = activeSnipes[JSON.stringify(channel)];
  if (typeof(snipe) === "undefined") {
    launchSnipe(channel);
    activeSnipes[JSON.stringify(channel)].participants.push({
      transaction: txn,
      username: txn.fromUsername,
    });
  }

  console.log("betting_open 178");
  if (snipe.betting_open === false) {
    bot.chat.send(channel, {
      body: `Betting has closed - refunding`,
    });
    processRefund(txn, channel);
    return;
  }

  activeSnipes[JSON.stringify(channel)].participants.push({
    transaction: txn,
    username: txn.fromUsername,
  });
  bot.chat.send(channel, {
    body: `@${txn.fromUsername} is locked into the snipe!`,
  });
  resetSnipeClock(channel);
}

function resetSnipeClock(channel: ChatChannel): void {

  const snipeTimeout: number = 60;
  clearTimeout(activeSnipes[JSON.stringify(channel)].timeout);
  bot.chat.delete(channel, activeSnipes[JSON.stringify(channel)].clock, {});
  bot.chat.send(channel, {
    body: `Betting stops in ${snipeTimeout} seconds`,
  }).then((sentMessage) => {
    runClock(channel, sentMessage.id, snipeTimeout);
    activeSnipes[JSON.stringify(channel)].clock = sentMessage.id;
  });
  const finalizeBetsTimeout: NodeJS.Timeout = setTimeout(() => {
    finalizeBets(channel);
  }, snipeTimeout * 1000);
  activeSnipes[JSON.stringify(channel)].timeout = finalizeBetsTimeout;

}

const activeSnipes: object = {};

function launchSnipe(channel: ChatChannel): void {
  // Tell the channel: OK, your snipe has been accepted for routing.

  const snipeTimeout: number = 60;
  let message: string = "The snipe is on.  Bet in multiples of 0.01XLM.  Betting format:";
  message += `\`\`\`+0.01XLM@${botUsername}\`\`\``;

  activeSnipes[JSON.stringify(channel)] = {
    betting_open: true,
    clock: null,
    participants: [],
    timeout: null,
  };

  bot.chat.send(channel, { body: message });

  bot.chat.send(channel, {
    body: `Betting stops in ${snipeTimeout} seconds`,
  }).then((sentMessage) => {
    runClock(channel, sentMessage.id, snipeTimeout);
    activeSnipes[JSON.stringify(channel)].clock = sentMessage.id;
  });

  const finalizeBetsTimeout: NodeJS.Timeout = setTimeout(() => {
    finalizeBets(channel);
  }, snipeTimeout * 1000);
  activeSnipes[JSON.stringify(channel)].timeout = finalizeBetsTimeout;

}

function finalizeBets(channel: ChatChannel): void {
  bot.chat.send(channel, {
    body: "No more bets!",
  });

  console.log("betting_open 255");
  activeSnipes[JSON.stringify(channel)].betting_open = false;
   // Give 5 seconds to finalize transactions + 1 extra.
  setTimeout(() => {
    executeFlipOrCancel(channel);
  }, 6 * 1000);
}

/* TODO: check that there are _different_ participants not someone betting against themself multiple times */
function executeFlipOrCancel(channel: ChatChannel): void {
  const snipe: ISnipe = activeSnipes[JSON.stringify(channel)];
  if (typeof(snipe) !== "undefined") {
    const participantUsernames: Array = snipe.participants.map((participant) => participant.username);
    const uniqParticipants: Array = _.union(participantUsernames);
    if (uniqParticipants.length > 1) {
      flip(channel);
    } else if (uniqParticipants.length === 1) {
      snipe.participants.forEach((participant) => {
        processRefund(participant.transaction, channel);
      });
      bot.chat.send(channel, {
        body: "The snipe has been cancelled due to a lack of participants.",
      });
      documentSnipe(channel, null, true);
      activeSnipes[JSON.stringify(channel)] = undefined;
    } else {
      bot.chat.send(channel, {
        body: "The snipe has been cancelled due to a lack of participants.",
      });
      documentSnipe(channel, null, true);
      activeSnipes[JSON.stringify(channel)] = undefined;
    }
  }
}

function cancelFlip(conversationId: string, channel: ChatChannel, err: Error): void {
  clearInterval(flipMonitorIntervals[conversationId]);
  if (typeof(activeSnipes[JSON.stringify(channel)]) !== "undefined") {
    bot.chat.send(channel, {
      body: `The flip has been cancelled due to error, and everyone is getting a refund`,
    });
    activeSnipes[JSON.stringify(channel)].participants.forEach((participant) => {
      processRefund(participant.transaction, channel);
    });
    documentSnipe(channel, null, true);
    activeSnipes[JSON.stringify(channel)] = undefined;
  }
}

const flipMonitorIntervals: object = {};

function monitorFlipResults(msg: MessageSummary): void {

  flipMonitorIntervals[msg.conversationId] = setInterval((() => {
    try {
      bot.chat.loadFlip(
        msg.conversationId,
        msg.content.flip.flipConvId,
        msg.id,
        msg.content.flip.gameId,
      ).then((flipDetails) => {
        if (flipDetails.phase === 2) {
          console.log("results are in");
          resolveFlip(msg.channel, flipDetails.resultInfo.number);
          clearInterval(flipMonitorIntervals[msg.conversationId]);
          activeSnipes[JSON.stringify(msg.channel)] = undefined;
        } else {
          console.log("results are NOT in", flipDetails);
        }
      }).catch((err) => {

        cancelFlip(msg.conversationId, msg.channel, err);

      });
    } catch (err) {
      cancelFlip(msg.conversationId, msg.channel, err);
    }
  }), 1000);
}

const allClocks: Array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].reverse();
const runningClocks: object = {};

function runClock(channel: ChatChannel, messageId: string, seconds: number): void {
  try {
    bot.chat.edit(channel, messageId, {
      message: {
        body: ":clock" + allClocks[seconds % 12].toString() + ":" + ` betting stops in ${seconds}s`,
      },
    });
  } catch (e) {
    return;
  }

  if (seconds > 1) {
    setTimeout(() => {
      runClock(channel, messageId, seconds - 1);
    }, 1000);
  } else {
    setTimeout(() => {
      bot.chat.edit(channel, messageId, {
        message: {
          body: "~:clock" + allClocks[seconds % 12].toString() + ":" + ` betting stops in 1s~ no longer accepting bets`,
        },
      });
    }, 1000);
  }
}

async function main(): Promise {
  try {
    await bot.init(botUsername, paperkey);
    console.log(`Bot initialized with username ${botUsername}.`);
    await bot2.init(botUsername, paperkey2);
    console.log("Second key initialized");
    console.log("Listening for all messages...");

    const channel: object = {
       membersType: "team", name: "mkbot", public: false, topicName: "test3", topicType: "chat",
    };
    const message: object = {
      body: `${botUsername} was just restarted...[development mode] [use at own risk] [not functional]`,
    };

    bot.chat.send(channel, message);

    await bot.chat.watchAllChannelsForNewMessages(
      async (msg) => {
        try {
          if (msg.content.type === "flip" && msg.sender.username === botUsername) {
            monitorFlipResults(msg);
            return;
          }
          if (msg.content.type === "text" && msg.content.text.payments && msg.content.text.payments.length === 1) {
            extractTxn(msg);
          }
        } catch (err) {
          console.error(err);
        }
      },
      (e) => console.error(e),
    );
  } catch (error) {
    console.error(error);
  }
}

async function shutDown(): Promise {
  await bot.deinit();
  await bot2.deinit();
  process.exit();
}

process.on("SIGINT", shutDown);
process.on("SIGTERM", shutDown);

main();

You might notice that this is mostly written in TypeScript!  As the project grew to 500 lines of code, I found it a bit difficult to manage bugs.  Converting the JS into TypeScript was mostly really easy.  A few rounds of TSLint and ESHint and TypeScript compilation later, everything was pretty much smooth sailing.

Misc note: I found the Type system something a bit counterintuitive to grapple with, especially when dealing with JSON and partially undocumented APIs.  Fortunately, the keybase-bot project has been partially implemented in typescript and all the types were already prepared for me. Niiiiice.

1C88E460-A57B-4EA2-B2C5-2C6ECFA2DFB5
v1 of the bot was designed on pen and paper, and then redesigned around some limitations3 I ran into as well as UX conveniences that seemed obvious in hindsight.

A few notes on deploying the bot…

@dxb, the leader of the @mkbot Keybase channel and the most helpful person in the testing and iterative development process4, showed me how to use Docker to deploy the bot.  I actually hadn’t had an opportunity to use Docker in production, so it was super informative!

Here was my deployment process:

  1. apt-get install docker-compose
  2. Added my current logged in user to the docker group with
    usermod -aG docker
  3. Logged out and logged back in
  4. Created a new folder and included a file called
    Dockerfile

    Here’s the Dockerfile:

    FROM archlinux/base
    
    RUN pacman -Sy && \
        pacman -S --ask 20 --noconfirm \
        base \
        binutils \
        fakeroot \
        git \
        jq \
        gcc \
        tcl \
        expect \
        nodejs-lts-dubnium \
        npm \
        sudo \
        keybase
    
    RUN useradd -m keybase && \
        mkdir /keybase && \
        chmod 777 /keybase
    
    RUN echo 'keybase ALL=(ALL) NOPASSWD: ALL' | EDITOR='tee -a' visudo
    
    USER keybase
    WORKDIR /home/keybase
    
    ENV PATH=/home/keybase:$PATH
    
  5. In the same folder, created a file called
    docker-compose.yaml

    Here’s the contents:

    version: '3.2'
    services:
        croupier:
            image: keybase_dxb:latest
            environment:
                BOT_OWNER: zackburt
                CROUPIER_PAPERKEY_1: warrior laugh...redacted...
                CROUPIER_PAPERKEY_2: warrior laugh...redacted...
                MYSQL_HOST: croupier-cluster.redacted.us-east-1.rds.amazonaws.com
                MYSQL_USER: croupier
                MYSQL_PASSWORD: redacted
                MYSQL_DB: croupier
            entrypoint: /home/keybase/scripts/start.sh
            restart: always
            volumes:
              - type: volume
                source: 0_home
                target: /home/keybase
              - type: bind
                source: ./0_scripts
                target: /home/keybase/scripts
    
    volumes:
        0_home:
    

    Note: we use two paper keys.  Due to limitations of the keybase client, the Keybase paperkey that launched the flip doesn’t have the ability to read the flip, so we use another paperkey for that.

     

  6. Built the Docker file with
    docker build --no-cache -t keybase_dxb .
  7. Created a folder called 0_scripts/ and put these scripts in it, sendmoney.sh and start.sh.  Notice that line in the docker-compose.yaml where start.sh is automatically invoked once the container starts.  Here they are:
    1. sendmoney.sh
      #!/bin/bash
      
      KB_TEAM=$1
      KB_CHAN=$2
      KB_AMT=$3
      KB_RCPT=$4
      WORKING_DIR=$5
      HOME_DIR=$6
      KB_SPAWN="expect -c 'spawn ${WORKING_DIR}/keybase --home ${HOME_DIR} chat send --channel ${KB_TEAM} ${KB_CHAN} \"+${KB_AMT}XLM@${KB_RCPT}\" ; expect \"if you are sure\" ; send -- \"sendmoney\r\" ; expect eof'"
      eval $KB_SPAWN
      

      Note: the keybase client requires some interaction to send money in chat, so @dxb cooked up this little expect script.

      Note as well: by default, sendMoneyInChat() is not a method in the keybase-bot library.  So we hacked up keybase-bot.js to add this method to the Chat class that extends ClientBase:

      sendMoneyInChat(channel, team, amount, recipient) {
      
        console.log('sendMoneyInChat: 1');
        const child = child_process.spawn('/home/keybase/scripts/sendmoney.sh',
          [channel, team, amount, recipient, this._workingDir, this.homeDir]);
      
        child.stdout.on('data', (data) => {
          console.log(`stdout: ${data}`);
        });
      
        child.stderr.on('data', (data) => {
          console.log(`stderr: ${data}`);
        });
      
        child.on('close', (code) => {
          console.log(`child process exited with code ${code}`);
        });
      
        child.on('error', (err) => {
          console.log(`child process errored with err ${err}`);
        });
      
      }
      
    2. start.sh
      #!/bin/bash
      
      # start keybase service without kbfs and gui
      keybase oneshot --username croupier --paperkey "warrior laugh redacted.."
      run_keybase -fg
      
      # put the commands to run at startup here
      sudo pacman -Scc
      
      keybase chat send $BOT_OWNER "$(date) - starting bot"
      
      sudo npm install -g typescript
      killall keybase
      rm -rf croupier 
      git clone https://github.com/codeforcash/croupier.git
      cd croupier
      npm install
      tsc --lib es2015 index.ts
      node index.js
      
  8. chmod +x 0_scripts/start.sh

    and

    chmod +x 0_scripts/sendmoney.sh
  9. Started the container with
    docker-compose up croupier

You’ll notice that the Docker container automatically builds based on the source code in the GitHub repo.  In fact, @dxb has it rigged so that his bots restart automatically whenever there’s a new commit to the codebase… but that’s outside the scope of this article as I haven’t followed him to that level yet. 🙂

Want to try croupier yourself?  Join the @mkbot channel on Keybase and get a game going with +0.01XLM@croupier.  Or, invite @croupier into any Keybase team you’re in and start the game the same way.

Is this legal?

Let it be said: I’m not a lawyer.  And Croupier does not take a rake except arguably that refunds and winnings are issued less transaction fees (maybe rounded up a few stroops – à la the salami slicing scheme described in Office Space and Superman 3).

As far as I know, in New York, skill games are legal and new legislation continues to reify that.  Now, is Croupier a skill game, given outcomes are determined by the provably just randomness features implemented on Keybase?

I say: yes!

  • One has the ability to increase one’s chance of winning through adding additional bets (although this doesn’t affect the expected value of one’s outcome, since one’s odds are always directly proportional to one’s bet commitment).
  • Croupier is inherently a social bot, and bet sizing has to be done in a way that doesn’t draw the ire of one’s companions – nobody wants to be steamrolled by someone with an unfairly large bankroll.
  • If somebody jukes the odds at the last minute, one has the ability to force a draw – and a refund of everybody’s bets: if two users with the same paper keys subscribe to the chat channel, Keybase doesn’t certify the randomness. In practice, Croupier refunds all bets. Forcing a duplicate registration event is a strategy, even though there are social consequences to being a bad actor.

Footnotes:

[0] Hacker News: social media for elite/elitist technologists who claim to despise social media and yet maintain some presence on the site.

[1] The Keybase founders, Max and Chris, previously launched SparkNotes and then OKCupid.

So perhaps indeed they could make cryptography cross the chasm.

[2] @pzduniak, thoroughly polite all the way through my plentiful nagging, added flip support to the keybase-bot library.

[3] Keybase has this cool feature where users may request money, but as of writing there was no convenient API for accessing it.

[4] @dxb, @locknload, @jessecooper, and @max were especially helpful testers! 

[5] Code For Cash is recruiting a developer to join Keybase and fully own development of the Windows Keybase client.  

[6] Croupier’s namesake is Dan Egan from the HBO series Veep.

dan egan
Dan’s colleagues undermined him by telling his boss he would “make a good croupier” – not a political operator.

Unlocking the information that candidates really care about when considering a new position

Here at Code For Cash, we are a 3rd party tech recruiter, which means that we are basically running a funnel.

funnel purple bg.png

Based on my background as a growth and revenue engineer at Lookout (which my colleagues turned into a $1b+ company 🦄), I know that the recipe for success is systematically testing tactics for optimizing each segment of the funnel.

Here’s a table of some of our open campaigns:

Landing Page URL Visits Applications Conv Rate
https://codefor.cash/bermi-backend 1144 20 1.75%
https://codefor.cash/betterment-engineering-manager 291 11 3.78%
https://codefor.cash/clearbit-backend-engineer 117 5 4.27%
https://codefor.cash/clearbit-fullstack-engineer 69 5 7.25%
https://codefor.cash/keybase-windows-developer 128 0 0.00%
https://codefor.cash/leverdata 406 8 1.97%
https://codefor.cash/pcap-sr-dev 557 14 2.51%
https://codefor.cash/pinchme-frontend-engineer 155 17 10.97%
https://codefor.cash/pinchme-product-manager 38 1 2.63%
https://codefor.cash/seated-head-engagement 333 18 5.41%

As you can see, we are running a weighted average conversion rate of 3.06%– plenty of room for improvement.  Through tons of iteration, I have grinded landing page conversion rates up from 0.5% to 10%+ – it’s all about kaizen.  Improving, getting more traffic, talking to the customer base, rinse and repeat.  I know it’s possible to improve because I have the successful experiential reference points.

So we recently bore witness to the Twitter activity of @jenistyping who was early employee and Head of People at Lever, and we were impressed…the radar went off, the alarm bells in my head started ringing: we should be paying attention to this.  So we got in touch and Jen made a referral to Peoplism.  Liz Kofman-Burns did a quick (hey, we’re bootstrapping!) review of our processes and came up with the following recommendations regarding information that we could include on each of our landing pages:

  1. URM candidates (but everyone really) wants to know: Will I belong? Is this a place where I’ll be treated fairly? How do I know that?
    1. Some ways to answer this are: getting an inventory of the team’s personalities and interests; understanding whether the company has historically been promoting from within; checking out the company’s Glassdoor presence; measuring employee tenure by reviewing LinkedIn profiles of current employees.
  2. What’s the impact the company has on the world?  (An evidentiated huge priority for millennials).
  3. What colleagues are they going to be interacting with day to day, and who benefits from their work?  i.e. what is this role’s impact on their colleagues?
  4. What are the growth and learning opportunities for everyone in the company?
  5. What’s work of this position like day to day?
  6. What are they going to be doing in 6 months?  In a year?
  7. What specific skills and values are you looking for? What will candidates be evaluated on?
  8. What are the interview questions?
  9. What is the interview *process*?
  10. Clear delineation of absolute hard skill requirements vs. nice-to-have
  11. Any compensation hints or guidelines

So, step one of improving our conversion rate is rewriting the stories we’re telling on the landing page to include the aforementioned information.

A lot of our clients are startups, “reassembling the airplane in the air.”  So it’s not always possible to provide absolute clarity regarding these topics.  But we’re going to systematically provide as much information as we have, and when we’re limited, we’re going to be clear about that.

Another improvement that’s almost always an easy win is to give the landing pages a facelift from the design team – doing the work of design improvements shows people that you care.  Here’s a preview of what the new design looks like:

Jobs Details

And one of the secrets in the digital community is that your content is not your content – it’s the title and the thumbnail image.  So we’re going to be careful about what thumbnail images we provide our various landing pages.  This is the guide we’re following to keep a strong technical handle on how our preview images render. 

Yes, a well-run funnel is only as good as the quality and relevancy of the people entering… we are not gonna speak to our experiments regarding this yet.

Thanks for staying tuned.

Go Givers Sell More: Compressing a 160 page book into 4 pages of quick notes

My parents recently went traveling in Vietnam and Laos and they met a couple that owns a really successful recruiting agency. They said one book, called Go Givers Sell More, made a huge impact. Since I am building a recruiting agency I figured that I should read it.

The book really resonated with my experience, especially now that I am doing a ton of sales and cold outreach. Here are my notes. I did my best to compress a 160 page book into 4 pages of notes. So if anything is vague or cryptic just ask.

Here are my notes.

The basic premise of the book is that you have to give value in order to receive.

If your goal in sales is to create value for other people, how do you do that? Here are five ways: excellence, consistency, attention, empathy, and appreciation.

  • Excellence: When you thank an employee for something at an excellent hotel, the answer is My Pleasure! rather than no problem.
  • Consistency: Do what you say you are going to do.
  • Attention: Pay phenomenal attention to detail and exhibit thoughtfulness.
  • Empathy: Put yourself in the other person’s shoes and make their life easier.
  • Appreciation: Say thank you, and mean it. Write thank-you notes… not just emails, but actual handwritten notes. Publicly and privately praise what you truly appreciate in other people.

With these five value-generating activities, it’s not formulaic that if you perform X acts of appreciation you will get Y sales in return. You give because it’s who you are and therefore what you do. The only way to get more is to continuously give away these acts of value and eventually repayment will emerge. But if you go about creating value with the ulterior motive of receiving more value, it tends to show through on some level and sabotage the result. Give without emotional attachment to the return — knowing confidently that there will be return.

If someone approaches you to do business but we know for their purpose that a competitor would be better, we make a referral. Our focus is on providing value for the customer.

Psychology: in order to become givers, we willingly suspend our self-interest. We don’t erase or deny it; we simply set it aside for a moment so we can gain emotional access to the full experience of the Law of Value.

Providing more value than you receive in payment is the trade secret of all exceptional businesses.

Rapport formula: F-O-R-M. Family, Occupation, Recreation, Message. Conversations that meander along these topics will strike a vein of connection and rapport.

Quick rapport rules:

  • Be polite.
  • Don’t interrupt.
  • Listen. Just try to understand exactly what the person is saying. Honor their perspective.
  • Smile (again).
  • Say please and thank you.
  • Find a genuine interest in the other person.

Rapport can be as subtle as honoring a person’s native tongue or expressing your genuine mutual interests. It’s about being human.

Going to the trouble of getting someone’s name right is one of the simplest gestures of respect there is — and mispronouncing, mangling, or misremembering another’s name is one of the surest ways to offend.

Remember to keep your focus on providing value for the other person. When you feel uncomfortable on a call, you are thinking about yourself.

Emotional maturity is the ability to keep your focus on others’ feelings even as you acknowledge and honor your own.

The essence of professionalism, is showing up for work even when you don’t feel like it. Feelings and moods come and go. There may be times when you don’t really feel you’re interested in this other person, don’t feel like creating value for them, or don’t feel like being friendly. That’s okay. Take the action anyway.

Genuine influence accrues to those who become known as the sort of person who is committed to helping other people get what they want.

Approach each business relationship with; What have I done for you lately?

Give credit away rather than seek it; be kingmakers rather than kings; be constantly on the lookout for ways they can help to elevate other people’s lives.

All things being equal, people will do business with and refer business to those people they know, like, and trust. Key aspect of likability: great grooming. Pay attention to it daily, even when working remotely. It shines through in your voice and subtle mannerisms, even through text.

The majority of your best customers will come from people you vaguely know: not exactly friends but not exactly strangers. The teller at your bank; your kids’ friends’ moms, and so forth. Improve weak ties.

The philosophy for attending meetups: just making friends. Why? To make friends. Don’t mention our service at all. How to make the perfect meetup pitch? Don’t. Your aim is to have fun and make friends. Go for quality, not quantity. And don’t pitch. Instead, ask great questions:

  • How did you get started in the [_____] business?
  • What do you enjoy most about what you do?
  • {Name}, how can I know if someone I’m talking to would be a good connection for you?

Meetups must be accompanied by followthrough.

  • Send a personalized, handwritten thank-you note that says what a pleasure it was to meet them.
  • Connect them to other people and suggest ways they could do business with each other or benefit in some way from the relationship.
  • Send them info they might find interesting or valuable – not about your product or service but about something they’re personally interested in.

A feature is about our service. A benefit is about the other person.

If someone asks, what do you do? That describes the benefits people derive from doing business with you.

You cannot “make a sale”. What you can do is create value. People will do what people will do. All you can do is seek to serve, look for ways to create value… and trust.

Emotional clarity is your understanding of the difference between your economic need (which is real) and your emotional need for this person to be the solution to that economic need. Emotional discipline is your ability to hold onto that clarity and consistently choose your responses to each situation, rather than acting impulsively.

The combination of clarity and discipline is posture: stepping into the truth of who you are and the value you have to offer, without emotional attachment to any specific outcome.

There’s not a lot riding on the outcome for you… but there could be for them.

Whenever a prospect brings up your competitor, go out of your way to say something nice about him or her because respect earns respect.

Authenticity is not something you seek or take on; it’s something you simply embrace. Being whole means your words and actions are not separate: you do what you say.

Keep your focus on: Who is this person? What do they want? What are they searching for? What is the single most valuable thing I could possibly offer them?

Remember, it’s not about you; it’s about them.

There is forceful assertion and bravado (bad) and there is the simple statement of fact that springs from the quiet stillness of authenticity.

If someone objects to a pitch, turn in the direction of the skid. Agree with what they said. “Certainly something to consider” and “I get what you’re saying” are ways of being with the person rather than jousting. When you say “That’s a great question” or “That’s a good point” and then join with them to examine the issue they raised, you let them know it’s welcome and appreciated. The only way to say that authentically is to genuinely see it as a great point.

The underlying goal is to create value for the other person. Things we say to help take the pressure off, so they can feel more relaxed about making the right decision:

  • If you can’t do it, I’ll definitely understand.
  • {Name}, this may or may not be for you.

The flip side of giving value is letting yourself receive. If you don’t let yourself receive, you’re refusing the gifts of others — and you shut down the flow.

Focus on creating value in the world around you and for the people around you, and the greatest opportunities will come to you in moments and from places you never expected.

How do you get people to trust you? By being a trusting person. Living in trust means that having made your plan, you put it fully into action, investing it with excellence, consistency, attention, empathy, and appreciation.