While a growing body of literature finds positive impacts of Start-Up Subsidies (SUS) on labor market outcomes of participants, little is known about how the design of these programs shapes their effectiveness and hence how to improve policy. As experimental variation in program design is unavailable, we exploit the 2011 reform of the current German SUS program for the unemployed which strengthened case-workers' discretionary power, increased entry requirements and reduced monetary support. We estimate the impact of the reform on the program's effectiveness using samples of participants and non-participants from before and after the reform. To control for time-constant unobserved heterogeneity as well as differential selection patterns based on observable characteristics over time, we combine Difference-in-Differences with inverse probability weighting using covariate balancing propensity scores. Holding participants' observed characteristics as well as macroeconomic conditions constant, the results suggest that the reform was successful in raising employment effects on average. As these findings may be contaminated by changes in selection patterns based on unobserved characteristics, we assess our results using simulation-based sensitivity analyses and find that our estimates are highly robust to changes in unobserved characteristics. Hence, the reform most likely had a positive impact on the effectiveness of the program, suggesting that increasing entry requirements and reducing support increased the program's impacts while reducing the cost per participant.