• Context
  • NFT Neural Net Waifu Generator
    • Originally posted in Discord on 2021-11-16
    • https://discord.com/channels/@me/768273264136945674/910302594885369897
    • So obviously someone's done NFT gatcha already. But it's probably pretty shitty. Probably still room for NFT gatcha to be done well.
      • But gatcha and NFTs are sort of contradictory because traditional gatchas only have a few characters, while NFTs are unique. That could be solved by using semi-fungible tokens, or via procedural generation.
      • Procedural generation is how a lot of NFTs have been done so far, and they could be done better, but they still wouldn't be very impressive.
      • But as machine learning gets better, I think there might be better options than traditional procedural generation. Like, imagine this:
    • This Waifu Does Not Exist, but as a deeply-tagged NFT generator.
      • Individual waifus are generated by an image generation network trained on one of the Danbooru datasets.
      • But the waifu seed data is itself the NFT. The baseline waifu NFT is not a single image, it's the tagset plus seed that causes the network to generate that character.
      • Which means that each NFT has infinite possible images based off of it. The person who owns the underlying character right can call that right to pay additional resources to the generator algorithm, which then spins up a new image, also an NFT, based on the root character plus an additional tagset defining the new image.
      • The quality will originally be low, but if you can maintain backwards compatibility with the character seeds, you can upgrade the system and have the value of the new images continue going up.
      • But even better, if you set the cost ratios for additional purchases correctly, it's basically a money printer. You can have people buying up huge swaths of characters with no images to search through them for the best waifus they can sell off. You can have people in love with their particular character who buy tons of pictures of them. All sorts of profitable options.
      • Plus, it's actually somewhat algorithmically sound, and the ownership right actually means something, unlike with most current iterations.
• Outline
  • Concept
    • Buy a tag-defined OC, and hook into a system to automatically commission art and content about your OC from machine learning generators.
  • Governance Board
    • Project is managed by a board.
      • Possibly the board is a DAO, or some other fancy blockchain thing.
        • Doesn't really matter much.
    • The board maintains the systems, mints vouchers, and adds or updates generators.
      • This allows the project to improve over time.
      • Essential when machine learning is advancing so quickly and chaotically.
    • The board also collects profits.
      • Inspiration gratuity is taken from board profits, but allocation of inspiration gratuity does not convey voting authority.
  • Vouchers
    • The board mints vouchers.
    • Vouchers are identical and interchangeable.
    • Vouchers are auctioned to the highest bidder, at a rate determined by the board.
      • A neat thing to do would be to automatically mint new vouchers at a rate that adjusts to target a particular level of profit.
      • To reduce volatility and encourage long-term preregistration of accurate bids, it might be advisable to use a Vickrey auction.
        • https://en.wikipedia.org/wiki/Vickrey_auction
    • Vouchers can be freely traded, and are only useful for conversion to OCs.
  • OC Seeds
    • A voucher can be converted into an OC.
      • This involves selecting a set of desired tags and submitting them along with the authorization code contained in the voucher.
      • A script or smart contract provided by the board generates a character seed for a hypothetical character matching the included tags.
      • This seed can be used by generators to create content matching the character.
      • Converting a voucher to an OC costs a small fee, set by the board.
        • This fee should mostly cover processing costs, as profits should be centered instead on the vouchers and the generators.
    • The seed model is derived from analyzing clusters of content tagged as containing particular characters, and then extrapolating to create a similar space of possibilities for a new, hypothetical character.
      • Source content for this model and the generators could be taken from one of the Danbooru datasets, such as Danbooru2020.
        • https://www.gwern.net/Danbooru2020
    • An OC also contains a unique tag label, and a set of authentication keys representing the unique ownership of that OC.
      • It might be a good idea to allow subdivision of OC ownership, to allow collectively-owned OCs whose rights are controlled by vote.
      • Ownership of an OC token is explicitly a grant of copyright under US and international law, to the extent possible.
  • Generators
    • General
      • A generator is a script that takes an OC seed and generates art or content for the corresponding character.
      • A fee is charged for generating content, based on the cost of maintaining the generator.
      • Generated content is published on the blockchain, but can be privatized.
        • Content may be public, private, or available to any community member who owns an OC of their own.
      • When content is generated, copyright is assigned to the owner of the OC.
        • However, rights are reserved by the board to publish public or community content.
        • Additionally, rights are reserved by the board to include all generated content in generator models.
      • Generated content can be rated, and those ratings are included in the tags used in training models.
        • In particular, the generators utilize an owner rating and an average community rating to determine the quality of each machine-learning-generated instance.
    • Static Images
      • The static image generator uses a machine learning model like StyleGAN, trained on a dataset like Danbooru2020, to create unique anime-style images.
        • A pre-trained model like TADNE would also be a reasonable start.
          • https://aydao.ai/work/2021/01/18/stylegan2ext.html
      • This generator uses a seed derived from the OC seed to ensure that running the generator multiple times for the same OC will recognizably reproduce the same character each time.
      • Additional tags are provided to describe an image containing that character, and the generator utilizes a tag-aware image generation network to create an image which would have been tagged with the appropriate character and content tags.
      • The resulting image is released as an encrypted message on the network, with the keys distributed to the public, community, or just the creator and the board, per the decision of the OC owner.
        • Higher fees would be charged for each level of privacy, because this is a high-leverage decision and because greater publicity benefits the network.
      • Further, character tweaks could be provided akin to Waifu Labs, to allow for Fate-style character "flavors" with the same underlying appearance but significant color or style differences.
        • https://waifulabs.com/
      • A fee is charged based on the difficulty of the operation, including a share of training costs, plus a profit allocation.
    • Voice Synthesis
      • By cross-referencing a database of voice actors and their roles with the image generator's tagged character database, it should be possible to generate hypothetical voices from a set of images.
        • This could be based in projects like the Pony Preservation Project's attempts at generating complete voiceprints for MLP characters.
          • https://docs.google.com/document/d/1xe1Clvdg6EFFDtIkkFwT-NPLRDPvkV4G675SUKjxVRU/edit
      • This generated voiceprint could then be applied to existing machine learning voice synthesis, such as 15.ai or Astralite's pone.dev.
        • https://15.ai/
        • https://twitter.com/AstraliteHeart
    • Lip Syncing
      • Given even a single suitable image, a vtuber-esque articulated face model can be produced.
        • https://www.reddit.com/r/MachineLearning/comments/e1k092/rp_talking_head_anime_from_a_single_image/
      • Variants of methods like LipGAN or MakeItTalk can be used to map audio or text onto an image as lip syncing instructions.
        • https://paperswithcode.com/method/lipgan
        • https://github.com/yzhou359/MakeItTalk
      • Combined, these methods allow a character to visibly and audibly speak arbitrary text.
    • Further Development
      • Personalized Voice Assistants
        • With generation able to provide audio and lip syncing for arbitrary text, extensions could be provided for services like Mycroft, to allow voice assistants to sound like generated OCs.
          • https://mycroft.ai/
        • Even better would be to provide tone alteration.
          • Machine learning has been used for style analysis and conversion between writing styles of different authors.
            • https://www.cs.nmt.edu/~ramyaa/publications/ml_techniques_Stylometry.pdf
          • Style conversion could be used to swap default outputs from a voice assistant to sound more like the stylistic "voice" of a hypothetical character, again based on analysis of sets of, for instance, anime transcripts.
  • Fee Model
    • Cost of Operation
      • Any operation that occurs on the network charges a fee to pay for the cost of that operation.
        • Since blockchain networks are not very effective at machine learning computation, fees for blockchain-variants of these scripts are likely to either be trivial or excessive.
      • Off-chain resources will have to be priced by the board, but should still be priced relative to the cost of computation.
    • Pro Rata Training Costs
      • As training costs are a preliminary and background necessity to maintaining the system, these costs should be divided over the userbase.
        • A good target would be to aim to pay off the initial training fees over the first year of operations, once the rate of transactions is roughly established.
    • Profit
      • The board will take a profit allocation from the above operations costs.
        • A good target would be to aim for 10% of operation costs in additional profit on simpler operations like converting a voucher into an OC.
        • 10% is also appropriate for publicly-shared generated content.
        • Private content should charge more, such as 20% for community content and 30% for fully-private content.
      • Additionally, any profit beyond transaction fees on the minting of vouchers is to be taken by the board.
    • Inspiration Gratuity
      • Danbooru tags include artist tags, denoting the creator of the artwork.
        • This mean's it's possible to generate images in the style of a specified artist.
        • US law regards neural net processing as creating a derivative work, and mimicked artists have no right to compensation for their images being used in or mimicked by a network.
      • Despite this, it would be polite and potentially valuable to assign mimicked artists an inspiration gratuity.
        • An inspiration gratuity would be 10% of all profits collected on the generation of an item generated to purposefully mimic the style of a specific creator.
        • This gratuity would be evenly divided among all specified creators, if multiple are specified.
        • This payment will be set aside and marked for the relevant creators, and will count as a gift, rather than payment for any service or product.
        • Allocated portions will be public knowledge until a creator claims their share, at which point they may have future shares automatically transferred to them.
  • Automatic Dissolution
    • In the event that the board takes no actions for a period of one year, the project will be automatically dissolved.
    • Automatic dissolution publishes the keys encrypting the generator scripts and the OC seeds, allowing users to recreate all services for themselves.
    • Additionally, the inspiration gratuity fund will be automatically distributed pro rata by share of wealth participation in the network.
  • Technical Implementation
    • The OCGen project could be implemented on a wide variety of platforms.
    • However, it was designed to utilize the particular features of blockchain computing networks.
      • The most suitable blockchain computing networks are Ethereum, for its fame and maturity, or the Internet Computer, for its efficiency and scalability.
    • The intention is that board membership shall be represented by DAO ownership, vouchers shall be privately issued tokens, and OC seeds and generated content shall be published as NFTs.
      • Although, OCs should potentially have the ability to issue controlling shares, or even to split flavors of an OC off into separate NFTs.
    • As much as possible, it would be valuable to implement all these systems as smart contracts or blockchain apps.
      • This is unlikely to be efficient for generators, and almost certainly going to be unfeasible for training said generators.
      • Accordingly, the primary role of the board will be to manage this link between the smart contracts and the independently-hosted machine learning components.
        • Also to collect profits and extend the project, of course.