Add kCTF pow.py #1974

justinsteven · 2021-10-04T04:34:03Z

This adds a very thin wrapper around Google's kCTF pow.py (https://github.com/google/kctf/blob/v1/docker-images/challenge/pow.py)

This is a draft and a request for comments. It should not be merged just yet.

Demo:

[/pwd/pwntools]% python3 pwnlib/data/kctf/pow.py ask

Usage:
Solve pow: {} solve $challenge
Check pow: {} ask $difficulty
  $difficulty examples (for 1.6GHz CPU) in fast mode:
             1337:   1 sec
             31337:  30 secs
             313373: 5 mins

[/pwd/pwntools]% python3 pwnlib/data/kctf/pow.py ask 1337
[WARNING] kctf-pow using random.randrange() which is not cryptographically secure
== proof-of-work: enabled ==
please solve a pow first
You can run the solver with:
    python3 <(curl -sSL https://goo.gle/kctf-pow) solve s.AAU5.AACHSYfS7R9fjck4/AaU+qCG
===================

Solution? ^CTraceback (most recent call last):
  File "/pwd/pwntools/pwnlib/data/kctf/pow.py", line 202, in <module>
    main()
  File "/pwd/pwntools/pwnlib/data/kctf/pow.py", line 169, in main
    line = f.readline().decode("utf-8")
KeyboardInterrupt

[/pwd/pwntools]% python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pwn
>>> pwn.kctf_pow_<tab><tab>
pwn.kctf_pow_solve(   pwn.kctf_pow_verify(
>>> pwn.kctf_pow_solve("s.AAU5.AACHSYfS7R9fjck4/AaU+qCG")
[NOTICE] kctf-pow running 10x slower, gotta go fast? pip3 install gmpy2
's.AAAHmgGmZgtfk2AfN0JgnyLN5gcXdNtgt6eUSL89gadobDu0gHhJTDtS8jUJ2VGNJ/yqSB1Iu09qwX62VHe1Q1LXdUadsWekt9zj2tpG4FGoMUPsRMtQzVNTfTKglWY5D+d0ts5fhBccRu3kbljpn0pmVezFEWvdLeyZ2tRUAs7V1I260gRewCMl4Wt1zQU2ZGNZ1khpni60SU09lEARUl8h'
>>> pwn.kctf_pow_verify("s.AAU5.AACHSYfS7R9fjck4/AaU+qCG", _)
True

TODO:

Get pow.py working under Python2 or give up trying

Questions:

Where should this live? Should we create a new module, say pwnlib.pow?
Is it appropriate to be vendoring Google's pow.py? It is Apache 2.0 and I've adhered to the requirement of "You must cause any modified files to carry prominent notices stating that You changed the files"
- If it's not appropriate to be vendoring pow.py, please reject this PR. I can't find a PyPi package containing pow.py and don't want to make one.
- If it's appropriate to be vendoring pow.py where should it go? I'm assuming data is a bad place for it. Should a libraries or lib directory be created and referenced in LICENSE-pwntools.txt as being a place in which third-party code lives?
- If it's appropriate to be vendoring pow.py should some effort be made to hide it from consumers of pwn? The following behavior seems a bit unwanted to me.

% python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pwn
>>> pwn.data.<tab><tab>
pwn.data.absolute_import  pwn.data.kctf             pwn.data.path
pwn.data.elf              pwn.data.os
>>> pwn.data.kctf.pow.<tab><tab>
pwn.data.kctf.pow.CHALSIZE              pwn.data.kctf.pow.gmpy_sloth_square(
pwn.data.kctf.pow.GMP_NOTICE_ISSUED     pwn.data.kctf.pow.hashlib
pwn.data.kctf.pow.HAVE_GMP              pwn.data.kctf.pow.main(
pwn.data.kctf.pow.MODULUS               pwn.data.kctf.pow.os
pwn.data.kctf.pow.SOLVER_URL            pwn.data.kctf.pow.python_sloth_root(
pwn.data.kctf.pow.VERSION               pwn.data.kctf.pow.python_sloth_square(
pwn.data.kctf.pow.base64                pwn.data.kctf.pow.random
pwn.data.kctf.pow.can_bypass(           pwn.data.kctf.pow.sloth_root(
pwn.data.kctf.pow.decode_challenge(     pwn.data.kctf.pow.sloth_square(
pwn.data.kctf.pow.decode_number(        pwn.data.kctf.pow.socket
pwn.data.kctf.pow.encode_challenge(     pwn.data.kctf.pow.solve_challenge(
pwn.data.kctf.pow.encode_number(        pwn.data.kctf.pow.sys
pwn.data.kctf.pow.get_challenge(        pwn.data.kctf.pow.usage(
pwn.data.kctf.pow.gmpy_sloth_root(      pwn.data.kctf.pow.verify_challenge(

Arusekk · 2021-10-04T07:11:47Z

Is kctf pow on pypi? Maybe it would be better to include it from there, or to make them submit it to pypi... otherwise looks fine

justinsteven · 2021-10-04T07:24:37Z

Is kctf pow on pypi?

I couldn't find any implementations on there, granted I didn't look too hard. I've also come to learn that the canonical pow.py is not Python 2 compatible, and so even if someone did put it on PyPi it probably wouldn't work for us.

It's still TODO for me to make the vendored kctf/pow.py Python 2 compatible.

I think pwntools should consider dropping Python 2 compatibility one day soon, but that's another matter... I see it was most recently discussed on #1741.

What do you think of the use of data for holding kctf/pow.py as a library? Is data an ok place for it? (constants seemed like an even worse place)
Do you think misc is an OK place for this, or should we create a pow module to also hold things like Add pow.py for proof-of-work #1318 (if it gets reopened one day)

Edited to add: Just for absolute clarity, #1318's pow.py is different to kctf/pow.py. It's why I prefixed the new functions with kctf_

justinsteven · 2021-10-04T07:59:15Z

Added wrapper around challenge generator

% python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pwn
>>> pwn.kctf_pow_<tab><tab>
pwn.kctf_pow_generate_challenge(  pwn.kctf_pow_verify(
pwn.kctf_pow_solve(
>>> pwn.kctf_pow_ ^C
KeyboardInterrupt
>>> challenge = pwn.kctf_pow_generate_challenge(1337)
[WARNING] kctf-pow using random.randrange() which is not cryptographically secure
>>> challenge
's.AAU5.AACvAidE/gWv1+w9iNc0dqv2'
>>> pwn.kctf_pow_verify(challenge, pwn.kctf_pow_solve(challenge))
[NOTICE] kctf-pow running 10x slower, gotta go fast? pip3 install gmpy2
True

Also added a more complete test to pwnlib.util.misc.kctf_pow_solve() to ensure the changes to pow.py don't break compatibility with the canonical pow.py

justinsteven · 2021-10-05T08:16:31Z

Used the following to connect to a host which was running the canonical pow.py under Python 3. This was done to stress test the changes made to pow.py to ensure the changes don't break compatibility.

import pwn
import six
import sys

while True:
    with pwn.remote(sys.argv[1], sys.argv[2]) as t:
        t.recvuntil(b"python3 <(curl -sSL https://goo.gle/kctf-pow) solve ")
        challenge = six.ensure_str(t.recvline())
        solution = pwn.kctf_pow_solve(challenge)
        t.sendlineafter(six.ensure_binary("Solution? "), solution)
        assert t.recvall() == six.ensure_binary("Correct\n")

Ran this stress test from a host running Python 2, a host running Python 3, and a host running Python 3 with gmpy2 installed.

This reverts commit c66a8c6.

Also brings back the use of secrets.randbelow() if Python 3 is being used

justinsteven · 2021-10-07T08:49:08Z

Ready for review. I'm happy to squash the commits if that's preferred.

peace-maker · 2021-10-07T13:12:55Z

pwnlib/util/packing.py

@@ -165,7 +165,7 @@ def pack(number, word_size = None, endianness = None, sign = None, **kwargs):
 return b''.join(reversed(out))

 @LocalNoarchContext
-def unpack(data, word_size = None):
+def unpack(data, word_size = None, endianness = None, sign = None, **kwargs):


This doesn't work, since endianness and sign are both overwritten with the value from context below. There are more occations where the docs differ from the real signature too.

Ah right. I lifted it from pack() but I see it doesn't use the @LocalNoachContext pattern

The linter, and I think my IDE, was complaining about the use of unpack() with the unspecified argument endianness

Would it be most correct to do:

@LocalNoarchContext def unpack(data, word_size = None, **kwargs):

Edit: For reference the linter was saying:

> pwnlib/data/kctf/pow.py:101:11: E1123: Unexpected keyword argument 'endian' in function call (unexpected-keyword-arg)

If this is expected I'm happy to undo the change to unpack()

justinsteven · 2021-10-25T22:36:50Z

Is there anything I can do to get this across the line? In the meantime can this please be given the label hacktoberfest-accepted

justinsteven · 2022-01-18T09:03:14Z

I've realised that a new module requires changes to docs/ but I'm not sure how to do that. If someone can let me know what that change should look like I'm happy to make it.

Arusekk

This change is big. The proposed interface is ugly, making the change not particularly useful.

Pwntools is a CTF tool box. The preferred interface would be a function to 'de-pow' a tube, in a similar way that 'de-alarming' is done.

x = remote(...)
x.solve_kctf_pow()
x.sendline(b'EXPLOIT GOES HERE')
x.interactive()

A readme entry would be nice in such a case.

Do not copy code and then comment out 70% of it, hey!
Just copy the relevant functions, provide attribution where needed, make it nice&small and integrate it.

Arusekk · 2022-12-28T22:18:13Z

pwnlib/util/packing.py

@@ -165,7 +165,7 @@ def pack(number, word_size = None, endianness = None, sign = None, **kwargs):
 return b''.join(reversed(out))

 @LocalNoarchContext
-def unpack(data, word_size = None):
+def unpack(data, word_size = None, **kwargs):


Unnecessary change.

Suggested change

def unpack(data, word_size = None, **kwargs):

def unpack(data, word_size = None):

Arusekk · 2022-12-28T22:18:33Z

pwnlib/pow.py

+ Returns:
+ A string representing the challenge
+ """
+ return _kctf_get_challenge(difficulty)


Suggested change

return _kctf_get_challenge(difficulty)

return _kctf_get_challenge(difficulty)

NL at end of file

Arusekk · 2022-12-28T22:18:55Z

pwnlib/data/kctf/pow.py

+# import os
+# import secrets
+# import socket
+# import sys
+# import hashlib


Suggested change

# import os

# import secrets

# import socket

# import sys

# import hashlib

Arusekk · 2022-12-28T22:19:19Z

pwnlib/data/kctf/pow.py

+ HAVE_GMP = True
+except ImportError:
+ HAVE_GMP = False
+ # sys.stderr.write("[NOTICE] Running 10x slower, gotta go fast? pip3 install gmpy2\n")


Why even have commented-out code?

Suggested change

# sys.stderr.write("[NOTICE] Running 10x slower, gotta go fast? pip3 install gmpy2\n")

Arusekk · 2022-12-28T22:21:04Z

pwnlib/data/kctf/pow.py

+ # return int.from_bytes(base64.b64decode(bytes(enc, 'utf-8')), 'big')
+ return packing.unpack(base64.b64decode(six.ensure_binary(enc, 'utf-8')), 'all', endian='big')


Do not use that six thing if an explicit way exists, please...

Suggested change

# return int.from_bytes(base64.b64decode(bytes(enc, 'utf-8')), 'big')

return packing.unpack(base64.b64decode(six.ensure_binary(enc, 'utf-8')), 'all', endian='big')

return packing.unpack(base64.b64decode(enc.decode('utf-8')), 'all', endian='big')

Arusekk · 2022-12-28T22:27:01Z

pwnlib/data/kctf/pow.py

+#def can_bypass(chal, sol):
+# from ecdsa import VerifyingKey
+# from ecdsa.util import sigdecode_der
+# if not sol.startswith('b.'):
+# return False
+# sig = bytes.fromhex(sol[2:])
+# with open("/kctf/pow-bypass/pow-bypass-key-pub.pem", "r") as fd:
+# vk = VerifyingKey.from_pem(fd.read())
+# return vk.verify(signature=sig, data=bytes(chal, 'ascii'), hashfunc=hashlib.sha256, sigdecode=sigdecode_der)
+


Commented code again...

Suggested change

#def can_bypass(chal, sol):

# from ecdsa import VerifyingKey

# from ecdsa.util import sigdecode_der

# if not sol.startswith('b.'):

# return False

# sig = bytes.fromhex(sol[2:])

# with open("/kctf/pow-bypass/pow-bypass-key-pub.pem", "r") as fd:

# vk = VerifyingKey.from_pem(fd.read())

# return vk.verify(signature=sig, data=bytes(chal, 'ascii'), hashfunc=hashlib.sha256, sigdecode=sigdecode_der)

Arusekk · 2022-12-28T22:27:59Z

pwnlib/data/kctf/pow.py

+# def usage():
+# sys.stdout.write('Usage:\n')
+# sys.stdout.write('Solve pow: {} solve $challenge\n')
+# sys.stdout.write('Check pow: {} ask $difficulty\n')
+# sys.stdout.write(' $difficulty examples (for 1.6GHz CPU) in fast mode:\n')
+# sys.stdout.write(' 1337: 1 sec\n')
+# sys.stdout.write(' 31337: 30 secs\n')
+# sys.stdout.write(' 313373: 5 mins\n')
+# sys.stdout.flush()
+# sys.exit(1)
+
+# def main():
+# if len(sys.argv) != 3:
+# usage()
+# sys.exit(1)
+#
+# cmd = sys.argv[1]
+#
+# if cmd == 'ask':
+# difficulty = int(sys.argv[2])
+#
+# if difficulty == 0:
+# sys.stdout.write("== proof-of-work: disabled ==\n")
+# sys.exit(0)
+#
+#
+# challenge = get_challenge(difficulty)
+#
+# sys.stdout.write("== proof-of-work: enabled ==\n")
+# sys.stdout.write("please solve a pow first\n")
+# sys.stdout.write("You can run the solver with:\n")
+# sys.stdout.write(" python3 <(curl -sSL {}) solve {}\n".format(SOLVER_URL, challenge))
+# sys.stdout.write("===================\n")
+# sys.stdout.write("\n")
+# sys.stdout.write("Solution? ")
+# sys.stdout.flush()
+# solution = ''
+# with os.fdopen(0, "rb", 0) as f:
+# while not solution:
+# line = f.readline().decode("utf-8")
+# if not line:
+# sys.stdout.write("EOF")
+# sys.stdout.flush()
+# sys.exit(1)
+# solution = line.strip()
+#
+# if verify_challenge(challenge, solution):
+# sys.stdout.write("Correct\n")
+# sys.stdout.flush()
+# sys.exit(0)
+# else:
+# sys.stdout.write("Proof-of-work fail")
+# sys.stdout.flush()
+#
+# elif cmd == 'solve':
+# challenge = sys.argv[2]
+# solution = solve_challenge(challenge)
+#
+# if verify_challenge(challenge, solution, False):
+# sys.stderr.write("Solution: \n".format(solution))
+# sys.stderr.flush()
+# sys.stdout.write(solution)
+# sys.stdout.flush()
+# sys.stderr.write("\n")
+# sys.stderr.flush()
+# sys.exit(0)
+# else:
+# usage()
+#
+# sys.exit(1)
+#
+# if __name__ == "__main__":
+# main()


Suggested change

# def usage():

# sys.stdout.write('Usage:\n')

# sys.stdout.write('Solve pow: {} solve $challenge\n')

# sys.stdout.write('Check pow: {} ask $difficulty\n')

# sys.stdout.write(' $difficulty examples (for 1.6GHz CPU) in fast mode:\n')

# sys.stdout.write(' 1337: 1 sec\n')

# sys.stdout.write(' 31337: 30 secs\n')

# sys.stdout.write(' 313373: 5 mins\n')

# sys.stdout.flush()

# sys.exit(1)

# def main():

# if len(sys.argv) != 3:

# usage()

# sys.exit(1)

#

# cmd = sys.argv[1]

#

# if cmd == 'ask':

# difficulty = int(sys.argv[2])

#

# if difficulty == 0:

# sys.stdout.write("== proof-of-work: disabled ==\n")

# sys.exit(0)

#

#

# challenge = get_challenge(difficulty)

#

# sys.stdout.write("== proof-of-work: enabled ==\n")

# sys.stdout.write("please solve a pow first\n")

# sys.stdout.write("You can run the solver with:\n")

# sys.stdout.write(" python3 <(curl -sSL {}) solve {}\n".format(SOLVER_URL, challenge))

# sys.stdout.write("===================\n")

# sys.stdout.write("\n")

# sys.stdout.write("Solution? ")

# sys.stdout.flush()

# solution = ''

# with os.fdopen(0, "rb", 0) as f:

# while not solution:

# line = f.readline().decode("utf-8")

# if not line:

# sys.stdout.write("EOF")

# sys.stdout.flush()

# sys.exit(1)

# solution = line.strip()

#

# if verify_challenge(challenge, solution):

# sys.stdout.write("Correct\n")

# sys.stdout.flush()

# sys.exit(0)

# else:

# sys.stdout.write("Proof-of-work fail")

# sys.stdout.flush()

#

# elif cmd == 'solve':

# challenge = sys.argv[2]

# solution = solve_challenge(challenge)

#

# if verify_challenge(challenge, solution, False):

# sys.stderr.write("Solution: \n".format(solution))

# sys.stderr.flush()

# sys.stdout.write(solution)

# sys.stdout.flush()

# sys.stderr.write("\n")

# sys.stderr.flush()

# sys.exit(0)

# else:

# usage()

#

# sys.exit(1)

#

# if __name__ == "__main__":

# main()

peace-maker · 2023-07-07T12:39:07Z

@justinsteven sorry for the long silence! Are you willing to keep working on this? I could have used this a few times and was always annoyed this isn't merged yet.

I think having a pwnlib.pow module where other pow "providers" could live as well in the future would be great and having a tube.solve_pow("kctf") function to handle the pow for you. Stripping the implementation and keeping a notice at the top, that this is derived from kctf's pow.py + license should be fine for the kctf class.

justinsteven · 2023-07-10T23:45:57Z

@peace-maker I don't have time to work on this at the moment I'm afraid, please feel free to take over if you'd like :)

justinsteven added 2 commits October 4, 2021 13:14

Add kCTF Proof of Work solver

291d5f8

Replace secrets with random for Python2 compat

6969d41

justinsteven added 3 commits October 4, 2021 17:52

Eliminate dependency on ecdsa

3272b41

Add kCTF Proof of Work challenge generator

039b082

Add entry for kCTF Proof of Work features

a7e2918

Make pow.py Python 2 compatible

2a6ec96

Also added a more complete test to pwnlib.util.misc.kctf_pow_solve() to ensure the changes to pow.py don't break compatibility with the canonical pow.py

justinsteven added 7 commits October 5, 2021 18:33

Try to appease pylint

c66a8c6

Revert "Try to appease pylint"

8a70587

This reverts commit c66a8c6.

Fix signature of unpack to try to appease PyLint

c5c4672

Disable unused code

c8739ad

Move kctf-pow wrappers into a dedicated pow module

2dc7b37

Remove dead code, use pwnlib.logging for warnings

53b3f4a

Also brings back the use of secrets.randbelow() if Python 3 is being used

Add pow

f00df1e

justinsteven marked this pull request as ready for review October 7, 2021 08:48

peace-maker reviewed Oct 7, 2021

View reviewed changes

Fix unpack() signature

ec906ee

Arusekk force-pushed the feature/kctf-pow branch from 1cd29fa to ec906ee Compare August 8, 2022 10:41

Merge branch 'dev' into feature/kctf-pow

aba009c

Arusekk requested changes Dec 28, 2022

View reviewed changes

peace-maker added feature help-wanted labels Jul 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kCTF pow.py #1974

Add kCTF pow.py #1974

justinsteven commented Oct 4, 2021 •

edited

Arusekk commented Oct 4, 2021

justinsteven commented Oct 4, 2021 •

edited

justinsteven commented Oct 4, 2021

justinsteven commented Oct 5, 2021

justinsteven commented Oct 7, 2021

peace-maker Oct 7, 2021

justinsteven Oct 7, 2021 •

edited

justinsteven commented Oct 25, 2021

justinsteven commented Jan 18, 2022

Arusekk left a comment

Arusekk Dec 28, 2022

Arusekk Dec 28, 2022

Arusekk Dec 28, 2022

Arusekk Dec 28, 2022

Arusekk Dec 28, 2022

Arusekk Dec 28, 2022

Arusekk Dec 28, 2022

peace-maker commented Jul 7, 2023

justinsteven commented Jul 10, 2023

	def unpack(data, word_size = None, **kwargs):
	def unpack(data, word_size = None):

	return _kctf_get_challenge(difficulty)
	return _kctf_get_challenge(difficulty)

		# return int.from_bytes(base64.b64decode(bytes(enc, 'utf-8')), 'big')
		return packing.unpack(base64.b64decode(six.ensure_binary(enc, 'utf-8')), 'all', endian='big')

	# return int.from_bytes(base64.b64decode(bytes(enc, 'utf-8')), 'big')
	return packing.unpack(base64.b64decode(six.ensure_binary(enc, 'utf-8')), 'all', endian='big')
	return packing.unpack(base64.b64decode(enc.decode('utf-8')), 'all', endian='big')

Add kCTF pow.py #1974

Are you sure you want to change the base?

Add kCTF pow.py #1974

Conversation

justinsteven commented Oct 4, 2021 • edited

Arusekk commented Oct 4, 2021

justinsteven commented Oct 4, 2021 • edited

justinsteven commented Oct 4, 2021

justinsteven commented Oct 5, 2021

justinsteven commented Oct 7, 2021

Choose a reason for hiding this comment

justinsteven Oct 7, 2021 • edited

Choose a reason for hiding this comment

justinsteven commented Oct 25, 2021

justinsteven commented Jan 18, 2022

Arusekk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peace-maker commented Jul 7, 2023

justinsteven commented Jul 10, 2023

justinsteven commented Oct 4, 2021 •

edited

justinsteven commented Oct 4, 2021 •

edited

justinsteven Oct 7, 2021 •

edited